My CI/CD runners are not picking up a pending job

I’m seeing really long delays in our pool of shared CI runners picking up pending jobs. It seems this change in behaviour is recent (in the past day or so). Eventually the job does get executed but it’s taking 10 minutes or more for it to even get picked up. Nothing has changed in terms of our gitlab-ci.yml configuration so I don’t think this is a configuration issue.

I’m looking for any tips on how to narrow down the root cause and then what solutions we can apply.

According to our CI/CD Settings we are using Shared Runners with Available Shared Runners = 15. We have no other jobs running on the project at the moment. One job is pending and is not getting picked up after a long delay. This is on cloud-hosted gitlab.com.

Any tips on narrowing down the root cause are appreciated.

Thanks,
Denis

Hi @dleclair

if you keep an eye on the status page this will tell you when there are problems with the shared runners. The history shows this last couple of weeks have had a few incidents.

And a ongoing incident regarding Shared Runners today.

Thank you. The status page is showing that the incident is resolved now so I’ll keep on eye on it.

status page shows nothing today, but already 15 min and counting being stuck on pending

It’s showing an active incident for me, sometimes the status page takes a while to catch up…

true, the funny thing is that the moment it pops as active, the pipeline took off for me …anyway I guess need to consider there are some stability issues around runners these days…

Yes, you can use your own runners if you prefer.

seems like an active issue. may be residual issue from outage a couple days ago. issue from status page is still open: 2021-03-12: Shared CI runners degraded (#3925) · Issues · GitLab.com / GitLab Infrastructure Team / production · GitLab