Can I populate a variable depending on another (dotenv/post-pipeline-creation) variable?

This is a bit of an open-ended problem I’m trying to solve, but here’s a summary of what I’m trying to achieve:

• Some jobs in my CI pipeline use a custom Docker image in our container registry
• These jobs use a version tag to stay “pinned” to a particular image until they’re ready to be upgraded
• Sometimes the Docker image will be rebuilt and tagged with a MR-specific version tag
• I’d like jobs to be able to use the MR-tagged version if it exists and the regular tagged version if it doesn’t.

I have some jobs in a pipeline that depend on a Docker image in the project’s container registry. The jobs use a version tag to stay pinned until they can be migrated eg.:

component-a/build:
  stage:
    build
  variables:
    IMAGE_VERSION: 1
  image:
    "$CI_REGISTRY_IMAGE/components:$IMAGE_VERSION"
  script:
    - # etc ...

component-b/build:
  stage:
    build
  variables:
    IMAGE_VERSION: 2
  image:
    "$CI_REGISTRY_IMAGE/components:$IMAGE_VERSION"
  script:
    - # etc ...

Sometimes though, there will be changes to the Docker image itself, and the pipeline will also rebuild that. In this case, I run a Kaniko job to rebuild the image and tag it with an extra suffix, eg. instead of version 3, it would be 3-mr10 for changes in merge request !10. That part all works.

The problem I now have is - how can I have jobs that use the image tag 3 use 3-mrNN when it exists? In other words, if:

• a pipeline includes a job that updates this Docker image; and
• a job uses a version of the image that matches IMAGE_UPDATED_VERSION

…I want that job to use IMAGE_UPDATED_TAG (3-mr10) instead of just the version (3):

My first attempt used rules. Firstly, the image building job exported the variables IMAGE_UPDATED_VERSION and IMAGE_UPDATED_TAG via a dotenv file, eg.

IMAGE_UPDATED_VERSION=3
IMAGE_UPDATED_TAG=3-mr10

And the jobs:

component-a/build:
  stage:
    build
  needs:
    job: component-image-build
    artifacts: true
    optional: true
  variables:
    IMAGE_VERSION: 3
  rules:
    # Did the image building job run? Did it affect this job's image? If so, use
    # the rebuilt image.
    - if: $IMAGE_VERSION == $IMAGE_UPDATED_VERSION
      variables:
        IMAGE_TAG_TO_USE: $IMAGE_UPDATED_TAG
    - when: always
      variables:
        IMAGE_TAG_TO_USE: $IMAGE_VERSION
  image: "$CI_REGISTRY_IMAGE/components:$IMAGE_TAG_TO_USE"

But that won’t work, because rules are evaluated at pipeline creation. They never see the dotenv variables. But perhaps it illustrates what I’m aiming for: if MR !10 triggered the Kaniko job to update v3 of the components image, component/build will use components:3-mr10 instead of components:3.

My next thought was to have variable expansion within variable names, so eg.

• component-builder exports eg. IMAGE_TAG_SUFFIX_V3=-mr10
• component-a/build has a variable IMAGE_VERSION=3
• component-a/build uses that to construct another variable name: IMAGE_TAG_SUFFIX_V${IMAGE_VERSION} which evaluates to the actual string "IMAGE_TAG_SUFFIX_V3"
• component-a/build expands the variable with the name it just constructed, IMAGE_TAG_SUFFIX_V3. If it doesn’t exist it’s just empty: "". This gets put in eg. IMAGE_TAG_SUFFIX.
• component-a/build then uses the image $CI_REGISTRY_IMAGE/components:${IMAGE_VERSION}${IMAGE_TAG_SUFFIX}

But there’s no way to parameterise the name of a variable as far as I can see.

At this point I’m a bit stuck. It doesn’t seem like it should be impossible, since these variables can actually be seen and used for image selection. There’s just no way I can see to “switch” one variable using the value of another in a way that the image key can use.

Platform information: using gitlab.com (EE/SaaS) 16.3.0-pre.

Like rules what image will be used in a job is determined at pipeline creation. So even if you modify the variable it won’t be applied. Only option I can think of is to create dynamic child pipeline.

I would go with 2 files, .gitlab-ci.yml and .pipeline-template.yml.

In .gitlab-ci.yml you will have job that will determine what custom image version to use and other jobs that use other images. In .pipeline-template.yml you will have your jobs that use custom image version.

Content of .gitlab-ci.yml could be like

...other definitions you need...

determine image version:
  script:
    - <logic to figure out what image to use>
    - sed -ie 's/IMAGE_TAG_TO_USE/<your_image_version>/g' .pipeline-template.yml
  artifacts:
    paths:
      - .pipeline-template.yml

child-pipeline:
  trigger:
    include:
      - artifact: .pipeline-template.yml
        job: "determine image version"

Content of .pipeline-template.yml

component-a/build:
  image: "$CI_REGISTRY_IMAGE/components:IMAGE_TAG_TO_USE"
  script:
...

The idea is that the determine image version job will replace the IMAGE_TAG_TO_USE with the correct image tag and then it’s created as child pipeline.

Note: please be aware that child pipelines have some limitations, the most problematic is that you cannot use artifacts from them in parent pipelines or in MR widgets,

It seems like it’s entirely possible to use dotenv-exported variables from an earlier job to select the image and tag in a later job. Here’s a comment on a related issue where someone verifies it, and I’ve been doing this myself by writing out the variable name manually.

I’ve started looking into dynamic child pipelines, but it would be a substantial increase in complexity over Gitlab’s typical static configuration. Having to manage extra templates, run a job to populate them, the extra few minutes per component of CI time, the UI itself (the pipeline view) is considerably more complex… they’re all minor things on their own, but trade off against the benefits. Maybe I’m in denial about avoiding the work, but I’m hoping it’s possible in a static configuration.

(What I really want is something like Github’s composite actions that take inputs and provide outputs like a kind of CI-function, but I don’t think that’s an ambition of Gitlab’s.)