CI: Specifying artifact dependencies when using `parallel: matrix:`

I know that this topic is a little bit old, but I stumbled upon this topic when I was searching for the same problem :sweat_smile:

It’s not possible (yet) to use those matrix values to specify the job name inside the dependencies block.
The “solution” I found was creating the artifacts inside the same parent directory but with a different name for each job. Because we can use the matrix values (like $DEVICE and $OS, in your case) inside the job, I can create an artifact with a different name on each job.

Let me show an example:

.parallel-config:
  parallel:
    matrix:
      - TEAM_NAME: team-a
        ENVIRONMENT:
          - dev
          - production
      - TEAM_NAME: team-b
        ENVIRONMENT:
          - test
          - production
  environment:
    # Will populate the CI_ENVIRONMENT_NAME variable
    name: ${TEAM_NAME}/${ENVIRONMENT}
  variables:
    # CI_ENVIRONMENT_NAME is available if "environment:name" is set
    ARTIFACT_FILEPATH: "${CI_PROJECT_DIR}/artifacts/${CI_ENVIRONMENT_NAME}/my_artifact.txt"

job-1:
  extends:
    - .parallel-config
  script:
    # Create directory for storing the artifact
    - mkdir -p $(dirname "$ARTIFACT_FILEPATH")
    - echo "Hello world from $CI_ENVIRONMENT_NAME" > "$ARTIFACT_FILEPATH"
  artifacts:
    name: "$CI_JOB_NAME artifacts from $CI_PROJECT_NAME on $CI_COMMIT_REF_SLUG"
    paths:
      # Exports everything inside the artifacts parent directory
      - "${CI_PROJECT_DIR}/artifacts/*"

job-2:
  extends:
    - .parallel-config
  needs:
    # Make all job-1 jobs run first
    - job-1
  dependencies:
    # Caveat: it will download all the artifacts from all job-1 jobs
    - job-1
  script:
    # Because every artifact is in a different directory, we will be fine
    - cat "$ARTIFACT_FILEPATH"

I hope it helps someone until they decide to implement this feature.

Edit: there is an issue open about this → Dynamic matrix arguments in dependencies (#396845) · Issues · GitLab.org / GitLab · GitLab

Edit 2: I’ve created a repository with some sample pipeline config → Eduardo Saporski / GitLab CI Parallel Matrix Artifacts · GitLab

1 Like