Gitlab CI: How do I use the environment variable from one stage as needs:project ref in another

I have two jobs in the same project: job A and job B.

job A creates an environment variable EXTERNAL_PROJ_REF=some_tag and exports it through a .env file.

job B needs to download artifacts from an external_project and package them with other artifacts from the current project. I want to be able to dynamically choose the commit reference from which these external artifacts get downloaded. I am trying to use the environment variable EXTERNAL_PROJ_REF as the ref for external_project needed by job B.

job A:
  stage: build
  script:
    - echo "EXTERNAL_PROJ_REF=`./generate_variable.sh`" > build.env  # evaluates to EXTERNAL_PROJ_REF=some_tag
  artifacts:
    reports:
      dotenv: build.env

job B:
  stage: package
  script:
    - ./do_packaging_job.sh
  needs:
    - project: external_project
      ref: $EXTERNAL_PROJ_REF
      job: external_job
      artifacts: true

When I run this pipeline though, job B instantly fails with the following error:

This job depends on other jobs with expired/erased artifacts:

If I hardcode ref to some_tag, the job does not fail, and I can confirm the EXTERNAL_PROJ_REF is successfully passed to job B.

job B:
  stage: package
  script:
    - echo "Ref = $EXTERNAL_PROJ_REF"  # Correctly prints "Ref = some_tag"
    - ./do_packaging_job.sh
  needs:
    - project: external_project
      ref: some_tag
      job: external_job
      artifacts: true

However, when I have ref:$EXTERNAL_PROJ_REF, the pipeline fails. Can somebody tell me if I’m missing something?

1 Like

I think your first job should use paths not reports:

job A:
  stage: build
  script:
    - echo "EXTERNAL_PROJ_REF=`./generate_variable.sh`" > build.env  # evaluates to EXTERNAL_PROJ_REF=some_tag
  artifacts:
    paths:
      - build.env

reports are for when you want the results of a job (for example, a set of unit tests) to be available via the UI in the merge request or pipeline pages on GitLab. paths are for when you want a file to be passed down the pipeline to subsequent jobs.

1 Like

Thanks for the suggestion, @snim2. It didn’t work though; the problem still maintains.

Can you download the artifact?

1 Like

Yes, the artifacts get downloaded, but the environment variable still seems to be unavailable at the time I need it

I’m not sure if I’ve understood you correctly. Is this what you’re aiming for?

1 Like

No, not quite. What you’re showing me works fine. The variable is available in the script: section of job B.

What I would like is for the variable to be available in the needs: section of job B (which seems to get used before the scripts section. I would like to use the environment variable to tell job B what artifacts to download from an external project.

So:

job B:
  stage: package
  script:
    - whatever command(s) here
  # Artifacts from the needs section get downloaded before above script is run
  needs:
    - job: job A
      artifacts: true
      # Below is an external project from which I need artifacts to run this job
    - project: external_project
      ref: $ENVIRONMENT_VARIABLE  # This should specify what tag/commit to download artifacts from
      job: external_job
      artifacts: true

Ah, right! I really think that it’s highly unlikely that that would be possible.

The more usual way to do this is to use a multi-project pipeline and so the “external” project in your example would have its pipeline run first, and then it would trigger job B and receive the right artifacts automatically.

Is that a possibility in your situation?

I’ll see if I can restructure the pipeline(s) to make this possible

1 Like

I finally realized Gitlab does not support what I want to do, at least not this way. According to this link, a variable passed from a different job can only be used in before_script, script or after_script sections of a job; it cannot be used to configure jobs. I cannot use it the needs section of job B.

Luckily, I found a simple workaround using the Gitlab API. I have API access to external_project, so I just use wget to download the artifact I need from the dynamically selected commit reference. Afterwards, I directly pass the artifact to job B.

job A:
  stage: build
  script:
    # Dynamically create commit reference
    - EXTERNAL_PROJ_REF=`./generate_commit_ref.sh`
    # Download artifact with Gitlab API
    - wget --header "PRIVATE-TOKEN:${EXTERNAL_PROJ_TOKEN}" --output-document outputFileName.file "${CI_API_V4_URL}/projects/${EXTERNAL_PROJ_ID}/jobs/artifacts/${EXTERNAL_PROJ_REF}/raw/${EXTERNAL_ARTIFACT_PATH}?job=${EXTERNAL_JOB_NAME}"
    # btw CI_API_V4_URL is a gitlab defined variable
  artifacts:
    paths:
      - outputFileName.file

job B:
  stage: package
  script:
    - ./do_packaging_job.sh
  needs:
    # Now this packaging job only needs job A. It doesn't need the external job anymore
    - job: job A
      artifacts: true
1 Like

Ah right. The other option is to dump the variable to a file in the first job, then read it back in the next job.