Unable to retrieve the needed artifacts in a child job if the artifacts are created in a parallel matrix

I have the following in .gitlab-ci.yml

  - release

  image: alpine:3.18
  stage: release
      - REPO: [prod]
      - ./${REPO}.txt
    - echo "this is ${REPO}" > ${REPO}.txt

  stage: release
    - release-new-version
    include: prod.yml
    strategy: depend

and I have the following in prod.yml

  image: alpine:3.18
    - pipeline: $PARENT_PIPELINE_ID
      job: release-new-version
    - apk add --update npm git
    - ls -la
    - cat version.txt
    - cat prod.txt

When I run this pipeline I can see that my artifact is created, but the triggered child pipeline is failing with This job could not start because it could not retrieve the needed artifacts.

If I do not use a parallel matrix then this works, however, I will need to expand this pipeline to multiple stages, and I will need the matrix.

Is this a big or should it be possible?


1 Like

You can check it out in official docs. Copied for reference:

  image: ruby:${RUBY_VERSION}
      - RUBY_VERSION: ["2.5", "2.6", "2.7", "3.0", "3.1"]
        PROVIDER: [aws, gcp]
  script: bundle install

  image: ruby:2.7
  stage: deploy
    - "ruby: [2.7, aws]"
  script: echo hello
  environment: production

I tried the above, but my dependancy is in a child pipeline, and I got the following error

jobs:deploy needs corresponding to dependencies must be from the same pipeline

Which i assume means that the deploy jobs needs to be in the same gitlab-ci.yml as the dependancy. Which it isn’t.

Ah, I missed the fact that it is from child pipeline. Pulling artifacts from child pipelines is not supported in general (not just for matrix jobs).
There are couple of Feature requests around this functionality, but there is no due date, yet.

Ref: Allow job in upstream pipeline to reference artifacts from a downstream pipeline (#285100) · Issues · GitLab.org / GitLab · GitLab and Seamless artifact handling for child/parent pipelines MVC (&4019) · Epics · GitLab.org · GitLab and Read child pipeline artifacts for MR reports (&8205) · Epics · GitLab.org · GitLab

@balonik I think there is some confusion - the scenario is a job in a downstream pipeline trying to pull an artefact from a upstream pipeline, not a job in the upstream pipeline trying to pull an artefact from a downstream pipeline.

The use case of a job in a downstream pipeline trying to pull an artefact from a upstream pipeline is documented at CI/CD YAML syntax reference | GitLab. So the issue is that this doesn’t support matrix jobs (I assume that this is because a matrix job is actually multiple jobs, each with a generated name). So to workaround this limitation, we need a single job in the upstream pipeline that aggregates the artefacts from all matrix jobs and then the downstream job needs the artefacts from the aggregating job (e.g. CI: Specifying artifact dependencies when using `parallel: matrix:` - #6 by esaporski)