Unable to retrieve the needed artifacts in a child job if the artifacts are created in a parallel matrix

GitLab CI/CD
1

I have the following in .gitlab-ci.yml

stages:
  - release

release-new-version:
  image: alpine:3.18
  stage: release
  parallel:
    matrix:
      - REPO: [prod]
  artifacts:
    paths:
      - ./${REPO}.txt
  script:
    - echo "this is ${REPO}" > ${REPO}.txt

deploy:prod:
  stage: release
  needs: 
    - release-new-version
  variables:
    PARENT_PIPELINE_ID: $CI_PIPELINE_ID
  trigger:
    include: prod.yml
    strategy: depend

and I have the following in prod.yml

release_job:
  image: alpine:3.18
  needs:
    - pipeline: $PARENT_PIPELINE_ID
      job: release-new-version
  script:
    - apk add --update npm git
    - ls -la
    - cat version.txt
    - cat prod.txt

When I run this pipeline I can see that my artifact is created, but the triggered child pipeline is failing with This job could not start because it could not retrieve the needed artifacts.

If I do not use a parallel matrix then this works, however, I will need to expand this pipeline to multiple stages, and I will need the matrix.

Is this a big or should it be possible?

Thanks

1 Like
2

You can check it out in official docs. Copied for reference:

ruby:
  image: ruby:${RUBY_VERSION}
  parallel:
    matrix:
      - RUBY_VERSION: ["2.5", "2.6", "2.7", "3.0", "3.1"]
        PROVIDER: [aws, gcp]
  script: bundle install

deploy:
  image: ruby:2.7
  stage: deploy
  dependencies:
    - "ruby: [2.7, aws]"
  script: echo hello
  environment: production
3

I tried the above, but my dependancy is in a child pipeline, and I got the following error

jobs:deploy needs corresponding to dependencies must be from the same pipeline

Which i assume means that the deploy jobs needs to be in the same gitlab-ci.yml as the dependancy. Which it isn’t.

4

Ah, I missed the fact that it is from child pipeline. Pulling artifacts from child pipelines is not supported in general (not just for matrix jobs).
There are couple of Feature requests around this functionality, but there is no due date, yet.

Ref: Allow job in upstream pipeline to reference artifacts from a downstream pipeline (#285100) · Issues · GitLab.org / GitLab · GitLab and Seamless artifact handling for child/parent pipelines MVC (&4019) · Epics · GitLab.org · GitLab and Read child pipeline artifacts for MR reports (&8205) · Epics · GitLab.org · GitLab

5

@balonik I think there is some confusion - the scenario is a job in a downstream pipeline trying to pull an artefact from a upstream pipeline, not a job in the upstream pipeline trying to pull an artefact from a downstream pipeline.

The use case of a job in a downstream pipeline trying to pull an artefact from a upstream pipeline is documented at CI/CD YAML syntax reference | GitLab. So the issue is that this doesn’t support matrix jobs (I assume that this is because a matrix job is actually multiple jobs, each with a generated name). So to workaround this limitation, we need a single job in the upstream pipeline that aggregates the artefacts from all matrix jobs and then the downstream job needs the artefacts from the aggregating job (e.g. CI: Specifying artifact dependencies when using `parallel: matrix:` - #6 by esaporski)