Inconsistent variable behavior in multi-project pipeline

I’m seeing a strange behavior in the variables passed in my multi-project pipeline and I’m not sure if it’s a bug or a feature. I hope that someone here can clarify what’s going on.

I’m trying to pass down a URL and a filename to the downstream and I’m getting inconsistent results, depending on what I do.

In my upstream gitlab-ci.yml:

variables:
  VERSION: "0.0.$CI_PIPELINE_IID"
  FILE: "${CI_PROJECT_NAME}-${VERSION}.tar.gz"
  URL: "${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/packages/generic/${CI_PROJECT_NAME}/${VERSION}/${FILE}"

stages:
  - deploy

test_vars:
  stage: deploy
  image: bash:latest
  script:
    - echo $URL
    - echo $FILE

trigger-downstream:
  stage: deploy
  trigger:
      project: "my-group/downstream"

My downstream gitlab-ci.yml:

stages:
  - deploy

downstream-job:
  image: bash:latest
  stage: deploy
  environment: production
  script:
    - echo $FILE
    - echo $URL

The output of test_vars is what you expect. The output of downstream-job is somewhat unexpected.
FILE contains upstream project name and downstream pipeline IID.
URL contains upstream project ID, upstream project name (in the url), downstream pipeline IID (in the version), downstream project name (in the file) and upstream pipeline IID (in the file).

Now if that’s not strange enough, we can make it even stranger.
Change the following in the upstream gitlab-ci.yml:

trigger-downstream:
  stage: deploy
  variables:
    NEW_URL: $URL
    NEW_FILE: $FILE
  trigger:
      project: "my-group/downstream"

And change the corresponding variables in the downstream:

downstream-job:
  image: bash:latest
  stage: deploy
  environment: production
  script:
    - echo $NEW_FILE
    - echo $NEW_URL

Now the output from the downstream changes!

FILE contains the downstream project name and the upstream pipeline IID.
URL contains the downstream project ID, downstream project name (in the url), upstream pipeline IID (in the version), upstream project name (in the file) and the downstream pipeline IID (in the file)

This is exactly the opposite of before.

My conclusion (for now) is that variables are expanded either in the downstream or upstream, depending on whether they have been wrapped an even or odd number of times. Which makes no sense to me.

Can someone tell me what the expected behavior is and how to consistently pass these two variables to my downstream.

I’m running on a self-managed gitlab version 15.5.4 with runner 15.5.1

Variables from upstream pipelines are inherited by downstream pipelines by default. To disable this you can use trigger:forward or inherit.

What you see is expected, because variables are expanded at the time of use, not at the time of declaration. This means that the variable VERSION is passed to downstream pipeline as "0.0.$CI_PIPELINE_IID" and expanded at use. Predefined CI/CD variables are not passed to downstream jobs. So in the downstream pipeline at the time of use this is expanded to what you see.

You can read more about it here.

This is also the behavior I expected. But that’s not what’s happening. $CI_PIPELINE_IID is expanded to two different values in the downstream pipeline. And these are swapped in the second example.

I tried

forward:
   pipeline_variables: true

that made no difference.
I didn’t try inherit. I’ll give that a go.

After more testing I’m definitely in the “this is a bug” camp.
inherit and forward don’t change the outcome and I can’t see how they would help in this situation.
@balonik according to your link, in the section Pass a predefined variable it says to store it in another variable. This works, also for global variables. But when they get used in other variables the result is indeterminate.

To show the complete absurdity, please see this:

Upstream:

variables:
  PIPELINE_IID: $CI_PIPELINE_IID
  PROJECT_NAME: $CI_PROJECT_NAME
  PROJECT_ID: $CI_PROJECT_ID
  VERSION: "0.0.$PIPELINE_IID"
  FILE: "${PROJECT_NAME}-${VERSION}.tar.gz"
  URL: "${CI_API_V4_URL}/projects/${PROJECT_ID}/packages/generic/${PROJECT_NAME}/${VERSION}/${FILE}"

stages:
  - deploy

trigger-downstream:
  stage: deploy
  variables:
    NEW_URL: $URL
    NEW_FILE: $FILE
  trigger:
      project: "my-group/downstream"

Downstream:

stages:
  - deploy

downstream-job:
  image: bash:latest
  stage: deploy
  environment: production
  script:
    - echo $NEW_FILE
    - echo $FILE
    - echo $URL
    - echo $NEW_URL
    - echo $PIPELINE_IID
    - echo $PROJECT_NAME
    - echo $PROJECT_ID

Results:

$ echo $NEW_FILE
upstream-0.0.120.tar.gz
$ echo $FILE
downstream-0.0.73.tar.gz
$ echo $URL
https://git.domain.net/api/v4/projects/6/packages/generic/downstream/0.0.73/upstream-0.0.120.tar.gz
$ echo $NEW_URL
https://git.domain.net/api/v4/projects/23/packages/generic/upstream/0.0.120/downstream-0.0.73.tar.gz
$ echo $PIPELINE_IID
73
$ echo $PROJECT_NAME
upstream
$ echo $PROJECT_ID
23

I agree that unfortunately, variable expansion in GitLab Runner is far from perfect. What you need to understand that while using a variable in job’s script relies on variable expansion of the shell, in other places like this it is expanded by GitLab Runner only once. The expansion is not recursive.

Taking this into consideration the behavior is actually like this:

PIPELINE_IID: $CI_PIPELINE_IID            =6                        -> expanded and passed as "6"
PROJECT_NAME: $CI_PROJECT_NAME            =upstream                 -> expanded and passed as "upstream"
PROJECT_ID: $CI_PROJECT_ID                =41294449                 -> expanded and passed as "41294449"
VERSION: "0.0.$PIPELINE_IID"              =0.0.5                    -> expanded and passed as "0.0.$CI_PIPELINE_IID" which is later by shell expanded to "0.0.5"
FILE: "${PROJECT_NAME}-${VERSION}.tar.gz" =downstream-0.0.6.tar.gz  -> expanded and passed as "$CI_PROJECT_NAME-0.0.$PIPELINE_IID.tar.gz" which is later by shell expanded to "downstream-0.0.6.tar.gz"
URL: "${CI_API_V4_URL}/projects/${PROJECT_ID}/packages/generic/${PROJECT_NAME}/${VERSION}/${FILE}" 
-> expanded and passed as ".../projects/$CI_PROJECT_ID/packages/generic/$CI_PROJECT_NAME/0.0.$PIPELINE_IID/${PROJECT_NAME}-${VERSION}.tar.gz"
which is in downstream expanded to "https://gitlab.com/api/v4/projects/41294452/packages/generic/downstream/0.0.6/upstream-0.0.5.tar.gz"

NEW_PIPELINE_IID: $PIPELINE_IID           =5                        -> expanded and passed as "$CI_PIPELINE_IID" which is later by shell expanded to "5"
NEW_PROJECT_NAME: $PROJECT_NAME           =downstream               -> expanded and passed as "$CI_PROJECT_NAME" which is later by shell expanded to "downstream"
NEW_PROJECT_ID: $PROJECT_ID               =41294452                 -> expanded and passed as "$CI_PROJECT_ID" which is later by shell expanded to "41294452"
NEW_VERSION: $VERSION                     =0.0.6                    -> expanded and passed as "0.0.$PIPELINE_IID" which is later by shell expanded to "0.0.6"
NEW_FILE: $FILE                           =upstream-0.0.5.tar.gz    -> expanded and passed as "${PROJECT_NAME}-${VERSION}.tar.gz" which is later by shell expanded to "upstream-0.0.$CI_PIPELINE_IID.tar.gz" which is expanded to "upstream-0.0.5.tar.gz"
NEW_URL: $URL                             =https://gitlab.com/api/v4/projects/41294449/packages/generic/upstream/0.0.5/downstream-0.0.6.tar.gz -> you get the idea...

The way inherit and forward helps is to only forward variables not having other variables in them and define the variables again in your downstream.

That explanation makes sense of what I’m seeing.
I still can’t see how I can control this with inherit and forward. Could you give an example?
What I’m trying to achieve is to pass a filename and a URL to a downstream. I have several upstreams, each with their own filenames and URLs. Downstream has no knowledge of where it got triggered from.

The only way I see this working, is by passing

PIPELINE_IID: $CI_PIPELINE_IID
PROJECT_NAME: $CI_PROJECT_NAME
PROJECT_ID: $CI_PROJECT_ID

and then reconstructing the URL and Filename in the downstream. This will probably work, but it feels silly, to duplicate the config, when I already have the variables in the upstream. Just to get around GitLab Runner behavior.

Exactly like this. And with inherit and forward you can only forward the variables you need to reconstruct, not the $FILE or $URL. So you can define the variables in downstream with the same names. And if you can do that, you can also actually define the variables in a common variables.yaml file and use include to include it everywhere you need.

variables.yaml:

variables:
  FILE: "${PROJECT_NAME}-${VERSION}.tar.gz"
  URL: "${CI_API_V4_URL}/projects/${PROJECT_ID}/packages/generic/${PROJECT_NAME}/${VERSION}/${FILE}"

downstream .gitlab-ci.yml

include:
    - project: 'my-group/my-project'
      file: '/templates/variables.yml'

Adjust as required.
Yes, it’s all to get around GitLab Runner behavior, but what other option there is.

This works.
You should be able to get a diploma as Certified GitLab CI Wrangler

Hello how to specify predefined variable in downstream if there is two values and to pick the respective one for each. One build number from upstream and another is from current. it needs to take respective build number to download package from jfrog. how to specify when there is two values for $CI_PIPELINE_IID. If you need more information let me know

There are no two values of $CI_PIPELINE_IID, there is one in each pipeline
If you need to specify $CI_PIPELINE_IID of upstream pipeline you need to send it to downstream pipeline as some other variable.

trigger-downstream:
  variables:
    UPSTREAM_PIPELINE_IID: $CI_PIPELINE_IID
...

This was just a headache! Thanks for the help @balonik and @magnum

Another way of handling this, that is less confusing… but possibly more work is to eschew the unwrapping that gitlab does and do it in bash (which collapses to the end result).

Then you pass it as an artifact to the trigger job. That way you make 100% it is unwrapped on the parent, it will just take an extra job, or adding a bit to the after_script of a previous job.

previous-job:
  variables:
    ENV_FILE: .dotenv
  script:
    - echo 'job'
  after_script:
    - echo "UPSTREAM_PIPELINE_IID=$CI_PIPELINE_IID" | tee "$ENV_FILE" >> $GITLAB_ENV
    - echo "UPSTREAM_PROJECT_NAME=$CI_PROJECT_NAME" | tee -a "$ENV_FILE" >> $GITLAB_ENV
  artifacts:
    when: on_success
    expire_in: "1 hour"
    reports:
      dotenv: $ENV_FILE



trigger-job:
  dependencies:
    - previous-job
  variables:
    UPSTREAM_PIPELINE_IID: $UPSTREAM_PIPELINE_IID
    UPSTREAM_PROJECT_NAME: $UPSTREAM_PROJECT_NAME
  trigger:
    project: downstream/project
    strategy: depend