Variable precedence and variable re-assignment

Hi!

I am on a self-managed GitLab instance, version 15.4.0-ee

I am racking my brain trying to figure out how to do this in the best way. I have a pipeline that is currently being triggered in three separate ways. Manually, scheduled and by another pipeline (multi-project).

The problem I am trying to solve is to be able to have some default values defined in my pipeline that can also be overridden by a manual, schedule, or pipeline trigger and forwarded to a downstream pipeline.

As default variables I was hoping to be able to use prefilled global variables that also show up in the UI with the added benefit of making it easier to trigger it manually.

The pipeline YAML below is a simplified version to try to demonstrate the problem I am trying to solve.
I would like to be able to control the pipeline execution regardless of trigger method, i.e. I would like to be able to set variable values via the UI, schedule, or parent pipeline bridge job, and have those values being used in this pipeline and also the resulting downstream pipeline.

variables:
  VAR1:
    value: "<some value>"
    description: "<Some description>"
  VAR2:
    value: "<some value>"
    description: "<Some description>"

stages:
  - prepare
  - deploy

generate-pipeline:
  stage: prepare
  tags:
    - tiny
  script:
        python3 generate_pipeline.py
  artifacts:
    paths:
      - generated-pipeline.yml

deploy:
  stage: deploy
  variables:
    VAR1: $VAR1
  trigger:
    include:
      - artifact: generated-pipeline.yml
        job: generate-pipeline
    strategy: depend

I suspect the problem has to do with variable precedence. The following doesn’t work on a job level.

variables:
  VAR1: $VAR1

The above will result in VAR1 getting the value “$VAR1” in the downstream pipeline if the trigger didn’t set this variable when triggering the pipeline. The “default” value of the global variable VAR1 is not used. I assume this has to do with variables declared on a job level has higher precedence than variables declared on a global level?

I have also tried to not define the variables on the job level. VAR2 that is only defined on a global level in the example YAML will keep its initial “default” value also in the downstream pipeline. But if we try to override VAR2 in a trigger it will still have the value defined by the YAML since we don’t expose this variable in the deploy bridge job.

One way to solve this is by using the following configuration in the deploy bridge job:

forward:
  pipeline_variables: true

This will make the downstream pipeline inherit all variables from upstream pipelines. Even from a pipeline in another project. I learned the hard way that those variables will also override local variables in my downstream pipeline if the parent pipeline happen to define a variable with the same name.

To solve the problem with variable contamination between projects the following configuration can be added to the bridge job that triggers the pipeline.

inherit:
  variables: false

But this would also require everyone that wants to trigger my pipeline from one of their own to know that they need to add this configuration.

If this construct had worked, I think there would be no problem:

variables:
  VAR1: $VAR1

It almost feels a little like a bug. Especially since you can find this kind of information in the documentation: GitLab CI/CD variables | GitLab

When I saw this documentation, I initially thought that this kind of variable declaration was OK on all levels.

But there is another way to solve this without having to forward pipeline variables. And that is to use variables with different names. So instead of VAR1: $VAR1 you would have to do something like this:

variables:
  VAR1:
    value: "<some value>"
    description: "<Some description>"

deploy:
  stage: deploy
  variables:
    VAR1_INTERNAL: $VAR1

This will result in both VAR1 and VAR1_INTERNAL being available in the downstream pipeline with the same or different values depending on what has been defined in the trigger. The drawback with this is that it will be a little messier having multiple variables for the same thing with different names. Especially when you start using pipeline templates.

If anyone has any suggestion for how this kind of setup can be solved in a better way, I am all ears.