Skip to content

Inspecting Prometheus metrics for e2e runs

Metrics gathering during e2e runs

kcp internally exposes the same metrics as kube api-server. These are being gathered during e2e runs if the PROMETHEUS_URL variable is set. Here, any e2e spawned kcp server will publish a scraping configuration to prometheus for the given Prometheus URL environment variable.

For e2e tests, the above environment variable is respected by the following test servers:

  • when a shared kcp server is spawned using SharedKcpServer or PrivateKcpServer in the kcp.test.e2e.framework package.
  • cmd/sharded-test-server
  • cmd/test-server/kcp

Inspecting metrics of e2e runs


To inspect metrics from GitHub e2e test runs, download the metrics from the GitHub summary page:

GitHub summary page

And execute Prometheus locally to inspect metrics:

$ mkdir -p metrics && unzip -p | tar xvzf - -C metrics
$ prometheus --storage.tsdb.path=metrics --config.file /dev/null


To inspect e2e metrics from prow runs, just copy-paste the prow link from the pull request:

PR prow link

And paste it into PromeCIeus:


Once the Prometheus pod is ready, the provisioned link can be called to inspect gathered metrics. Alternatively, download the prometheus tarball from the gcsweb frontend locally and start a Prometheus instance as described above.

Collecting metrics locally

To collect metrics locally, a convenience script is available to download and execute a Prometheus instance. Calling ... ./hack/run-in-prometheus make ... will download, start, collect metrics, and stop Prometheus during the test run.


$ WHAT=./test/e2e/virtual/apiexport TEST_ARGS="-v -run TestAPIExportAuthorizers" ./hack/ make test-e2e
A local Prometheus can then be started to inspect gathered metrics:
$ ./hack/tools/prometheus --storage.tsdb.path=.prometheus_data --config.file /dev/null