Monitoring CI metrics - build time, queue length etc

Hi All

Is there any way to automate Monitoring of CI-related entities?

For example:

  • how long did the job take
  • how long did the whole pipeline take
  • how long did waiting in the queue take
  • how long was between git push and pipeline finish (Lead Time = waiting time + pipeline time)

We are undergoing serious redesign of our software architecture and want to see how this will affect (speed up) builds.

Also, we want to optimize the use of underlying Infrastructure - essentially having some hardware empirically find out the best configuration of VMs \ CPU cores \ concurrent runners etc - to achieve fast feedback to a developer if his\her changes have broken the pipeline.

I’ve started looking at bundled Prometheus, and it looks like it collects the following metrics:

  • ci_created_builds (which is always “No datapoints found” in my case)
  • ci_pending_builds
  • ci_running_builds
  • ci_stale_builds (always 0 in my case)

So, number of pending and running builds is great, but not enough.

I found this Prometheus exporter that calculates those metrics:

Well, I guess I can learn Ruby… and extend this exporter with whatever metrics I need from PosgreSQL.
But I’m hoping there’s an easier way to get those Build Time \ Lead Time metrics?
Gitlab API?

Did you ever find a solution for this @boiky ?

No, we just used standard metrics:

  • ci_pending_builds
  • ci_running_builds

would this be what you seek?

For anyone else stumbling on this, the merge linked above and other docs essentially boil down to fetching http://localhost:9252/metrics (on the runner box) and pulling out the metrics you need (or piping them into your monitoring/metrics platform). This won’t do everything listed in the request, but it gets quite close, and definitely gives you some meaningful information.

I don’t yet have a way to see the CI queue (as requested here: Show CI Runner queue in the UI to help users understand when things get bogged down (#20617) · Issues · / GitLab · GitLab)