CI: Specifying artifact dependencies when using `parallel: matrix:`

How to Use GitLab
1

If I have a job that runs in parallel and produces different artifacts on each run how do I specify the dependencies on a particular parallel job.

E.g.

build:
  script:
       - "echo $JOB > artifact.txt"
   parallel:
     matrix:
       - JOB: ["1", "2"]
   artifacts:
      paths: artifact.txt

deploy:
  dependencies:
     - build # but which one?
  script:
     - cat artifact.txt
1 Like
2

I was looking for exactly the same. I was able to use the calculated job name which should be something like this in your case:

dependencies: ["build: [1]"]

I couldn’t find anything in the docs related to this, maybe it’s worth to add some notes there too.

3

Thanks, I ran into this too. Submitted documentation in Document how to use CI `dependencies` and `parallel:matrix` together (!82734) · Merge requests · GitLab.org / GitLab · GitLab.

4

Hello, do you think that it’s possible to set all dependencies dynamically ?
Something like that

build_for_test:
  stage: setup
  script:
    - build "$DEVICE, OS=$OS" -output "$artifact"
  artifacts:
    name: "$artifact"
    paths:
      - "$artifact"
  parallel:
    matrix:
      - DEVICE: ["iPhone X", "iPhone 7"]
        OS: ["12.4", "15.4"]

test:
  stage: test
  needs: ["build_for_test: [$DEVICE, $OS]"]
  dependencies:
    - "build_for_test: [$DEVICE, $OS]"
  script:
    - test "$DEVICE, OS=$OS"
  parallel:
    matrix:
      - DEVICE: ["iPhone X", "iPhone 7"]
        OS: ["12.4", "15.4"]

Here is the expected result:

build_for_test: [iPhone X, 12.4] → test: [iPhone X, 12.4]
build_for_test: [iPhone X, 15.4] → test: [iPhone X, 15.4]
build_for_test: [iPhone 7, 12.4] → test: [iPhone 7, 12.4]
build_for_test: [iPhone 7, 15.4] → test: [iPhone 7, 15.4]

2 Likes
5

As far as I can tell, it is not possible.
I tested this in Windows.
image

image

6

I know that this topic is a little bit old, but I stumbled upon this topic when I was searching for the same problem :sweat_smile:

It’s not possible (yet) to use those matrix values to specify the job name inside the dependencies block.
The “solution” I found was creating the artifacts inside the same parent directory but with a different name for each job. Because we can use the matrix values (like $DEVICE and $OS, in your case) inside the job, I can create an artifact with a different name on each job.

Let me show an example:

.parallel-config:
  parallel:
    matrix:
      - TEAM_NAME: team-a
        ENVIRONMENT:
          - dev
          - production
      - TEAM_NAME: team-b
        ENVIRONMENT:
          - test
          - production
  environment:
    # Will populate the CI_ENVIRONMENT_NAME variable
    name: ${TEAM_NAME}/${ENVIRONMENT}
  variables:
    # CI_ENVIRONMENT_NAME is available if "environment:name" is set
    ARTIFACT_FILEPATH: "${CI_PROJECT_DIR}/artifacts/${CI_ENVIRONMENT_NAME}/my_artifact.txt"

job-1:
  extends:
    - .parallel-config
  script:
    # Create directory for storing the artifact
    - mkdir -p $(dirname "$ARTIFACT_FILEPATH")
    - echo "Hello world from $CI_ENVIRONMENT_NAME" > "$ARTIFACT_FILEPATH"
  artifacts:
    name: "$CI_JOB_NAME artifacts from $CI_PROJECT_NAME on $CI_COMMIT_REF_SLUG"
    paths:
      # Exports everything inside the artifacts parent directory
      - "${CI_PROJECT_DIR}/artifacts/*"

job-2:
  extends:
    - .parallel-config
  needs:
    # Make all job-1 jobs run first
    - job-1
  dependencies:
    # Caveat: it will download all the artifacts from all job-1 jobs
    - job-1
  script:
    # Because every artifact is in a different directory, we will be fine
    - cat "$ARTIFACT_FILEPATH"

I hope it helps someone until they decide to implement this feature.

Edit: there is an issue open about this → Dynamic matrix arguments in dependencies (#396845) · Issues · GitLab.org / GitLab · GitLab

Edit 2: I’ve created a repository with some sample pipeline config → Eduardo Saporski / GitLab CI Parallel Matrix Artifacts · GitLab

1 Like