Help to understand how variables are handled in multi level pipelines

GitLab CI/CD
,
1

I would like to get some help understanding how variables are handled in multilevel pipelines.

I currently have 3 pipelines where I in the first one declare a variable using $CI_COMMIT_SHORT_SHA and it is not working as I would expect:

  • In the first pipeline I declare a global variable TEST_RUN_ID: $CI_COMMIT_SHORT_SHA and then creating a namespace variable from it TCP_URL: $TEST_RUN_ID-test-url.
    Then I tried to send this variable using forward:pipeline_variables: true for the multilevel-pipeline.
    In the triggered pipeline it seems to not have resolved the variable and takes the CI_COMMIT_SHORT_SHA from the child pipeline, this is the same for the next triggering of the last pipeline there it uses its commit sha.

  • *What version are you on?
    Are using self-managed Controlled by our ops-team so do not have much insight here.

    • GitLab: GitLab Enterprise Edition v16.7.4-ee

  • *I’ll add a basic copy of what my issue is since I can’t share much from our project. *

    • First pipeline with “short_sha=2622f949” and debug logs shows
      namespace=end2end-test-2622f949-run, CI file below:
variables:
 TEST_RUN_ID: $CI_COMMIT_SHORT_SHA 
 tcp_url: $TEST_RUN_ID-tcp-url
 namespace: end2end-test-$TEST_RUN_ID-run

stages:
  - build
  - get-variables
  - end2end-before-test  

before-end2end-test-job:
  stage: end2end-before-test
  trigger:
    project: 'projectName'
    branch: feature/branchName
    strategy: depend
    forward:
      pipeline_variables: true
  variables:
    step: before_test
    namespace_deploy: $namespace
  needs:
    - job: get-variable-for-test-job #Job that exists before this one.
      artifacts: true
  • Second pipeline with “short_sha=fbcf402c” and debug logs shows
    namespace=end2end-test-fbcf402c-run, CI file below:
stages:
  - end2end-test

create-namespace-job:
  stage: end2end-test
  rules:
    - if: $step == "before_test"
  script: #just a workaround to set the variable. Currently what makes the pipeline work.
    - echo "namespace_kafka=$namespace_deploy" >> namespaceForKafka.env
  artifacts:
    reports:
      dotenv: namespaceForKafka.env

kafka-before-test-job:
  stage: end2end-test
  rules:
    - if: $step == "before_test"
  trigger:
    project: 'projectName'
    branch: feature/branchName
    strategy: depend
    forward:
      pipeline_variables: true
  variables:
    step: $step
    namespace_kafka: "${namespace_kafka}" 
  needs:
    - job: create-namespace-job 
      artifacts: true
  • Third pipeline with “short_sha=c879477e” and debug logs shows
    namespace=end2end-test-fbcf402c-run, CI file below:
variables:
  GIT_STRATEGY: clone

stages:
  - deploy-kafka-end2end-test

before-test-job:
  stage: deploy-kafka-end2end-test
  rules:
    - if: $step == "before_test"
  script:
    - kubectl apply -n $namespace_kafka -k ./overlays/endtoend/
  • Have been reading Downstream pipelines | GitLab Docs and GitLab CI/CD variables | GitLab Docs to try to understand how variables are created and passed between pipelines. Was expecting the variable I set in the first job in the first pipeline to be the namespace that I could access in all pipelines. But to make my pipeline work I have been needing to pass it through what is shown above.

B.R.
Tore