I have a multi-stage pipeline to create some Docker images based on some variables. Each state needs access to these variables. For a single set of variables this is easy, because I can just specify them as top-level variables in my .gitlab-ci.yml file. But now I would like to run this pipeline for multiple sets of variables in parallel to create multiple sets of Docker images.
I have already created a working solution using the parallel matrix feature, but I’m not 100% happy with it, because I am duplicating the parallel matrix variables for each job, which looks something like this:
stages:
- stage1
- stage2
job1:
stage: stage1
script: job1-command "$VAR"
parallel:
matrix:
- VAR: ["value1", "value2"]
job2:
stage: stage2
script: job2-command "$VAR"
parallel:
matrix:
- VAR: ["value1", "value2"]
(My actual pipeline has more stages, more jobs and more variables.)
Apart from the duplication of the parallel matrix variables for each job another downside is that all stage1 jobs need to finish before a stage2 job can start. Ideally I would like job2 for value1 to start when job1 for value1 is done, without needing to wait for job1 for value2.
I thought I might be able to use the dotenv feature to only set the variables on the first job and then pass them down the pipeline, but I couldn’t figure out how to let a subsequent job depend on a particular dotenv for a previous parallel matrix job.
I think what I would really want is to be able to run two parallel instances of the whole pipeline with different variables as if I could specify parallel matrix variables on a pipeline level instead of on a job. Does GitLab CI have a mechanism to do this?
I’m currently using GitLab 13.7.9-ee.