When there are multiple pending pipelines the runners pick them up in either random order or the most recent pipeline is picked up first.
It is fundamental in my setup that the pipelines are executed in the order they were created.
Is there a way to ensure that with Gitlab CI/CD?
Are these pipelines in the same project?
Yes, they are in the same project.
If you are using your own runners, and you have only one GitLab runner for your project, with no concurrency; none of your jobs are interruptible and you do not have skip outdated deployments set, then I believe your pipelines should run sequentially.
You haven’t said much about your use-case though, and it is a unusual to need to control pipeline execution like this, and I think most people would say it is not desirable. For example, if I have 3 MRs open, with pipelines running on 3 different branches for those MRs, I would not expect (or want) to be able to control the order in which the 3 branch pipelines are run. Once I merge those three branches into my default branch, I might be a little more picky, because the default branch might be creating a deployment. I’m not sure whether that helps you though, because I don’t understand your use case.
Mine use case is a bit specific because we are migrating from a legacy CI system into GitLab and at least for the time being we need to replicate a few things that we used to have with the old system. One of these is a concept of build numbers. We generate a unique BUILD_NUMBER for each pipeline using script similar to https://gitlab.com/jmarcos.cano/CI_PIPELINE_IID/snippets/1675434. For various reasons using CI_PIPELINE_IID is not sufficient for our needs for now.
We generate BUILD_NUMER as the first stage (assign_build_number) of a pipeline execution. When pipelines are executed out of order the build numbers also get reshuffled and that’s a pain. Please see the below image showing pipelines together with the build numbers assigned to them. The order of the build numbers is equivalent to the order in which the pipelines were executed. Please not that this is not the order in which the pipelines were created.
The job in the mentioned assign_build_number stage has resource_group assigned to it to ensure no more than 1 instance is running at the same time.
We run our jobs at a shared GitLab runner. Jobs aren’t interruptible and we don’t have the skip outdated deployment set.
I’m okay with sharing more details in a private manner.
Hi @michal67 this is interesting!
I suspect that in the future, when your migration is more settled, you will want to relax the rules about out of order (OOO) execution.
One thing you might like to consider, rather than pinning the execution order (which you cannot do with shared runners) is to change the build number to rely on something like the date of the last commit. As you mentioned, CI_PIPELINE_IID is an ID that is incremented on your project, every time a pipeline starts running, so if you allow OOO execution, you cannot rely on that for a build number, but you could possibly find something else. The full list of pre-defined variables might help you, if you want to try something like this.
If this situation is only temporary whilst you migrate, and you don’t mind a bit of a kludge, I would be tempted to keep the build number in a file that is committed to your repo. Then that first stage would read the file, increment the number, commit it back to the repo and pass to the next stage.
