I’m using the SaaS version of GitLab.
I have a pipeline that spawns a child pipeline, and a private gitlab runner. I haven’t changed any of my local configuration in weeks if not months. My concurrency on the runner is 4.
Up until now, I’ve been able to spawn 100+ pipelines and have them run such that the oldest waiting job seems to run next. Therefore for a given pipeline, the child pipeline is spawned in a timely manner and runs as expected. Its jobs interleave with the jobs from the primary and child pipelines of the other pipelines, to a maximum of 4 concurrent jobs. No matter how many pipelines I spawn concurrently, they all go through eventually, and runner utilisation is high.
But in the last few days, the behaviour seems to have changed - my child pipelines do not spawn until all the primary pipelines have completed. But because my primary pipelines depend on my child pipelines, if I run more than 4 at a time, it all ends up in temporary deadlock until every single primary pipeline has completed. All the primary pipelines end up sitting there waiting for their child pipelines, and the child pipelines sit there waiting for an available runner. This makes the whole thing take significantly longer.
What’s going on?