Artifacts created but not downloaded by dependent job

Description

We have a three stage pipeline that uses child pipelines. The jobs are completely dependent on artifacts being passed to each subsequent stage:

  • checkout
  • build
  • test

We have two gitlab runner hosts configured. The first does most of the jobs while the second only runs jobs tagged vcan. Both are hosted in the same AWS VPC and both have the same Network and firewall configuration.

The artifacts are created correctly by the checkout stage:

Uploading artifacts for successful job
00:40
Uploading artifacts...
my_project: found 7757 matching files                     
Uploading artifacts to coordinator... ok            id=11010011 responseStatus=201 Created token=xxxxxxxx
Job succeeded

And downloaded by the build stage which then successfully creates it’s own artifacts:

Getting source from Git repository
00:02
Skipping Git repository setup
Skipping Git checkout
Skipping Git submodules setup
Restoring cache
00:02
Downloading artifacts
00:02
Running before_script and script

<snip>

Uploading artifacts for successful job
00:04
Uploading artifacts...
my_project/log: found 2 matching files                    
my_project/install: found 2 matching files                
my_project/build: found 2 matching files                  
my_project/src: found 2 matching files                    
my_project/../my_project*.deb: found 1 matching files                       
Uploading artifacts to coordinator... ok            id=110001100 responseStatus=201 Created token=xxxxxx
Job succeeded

But the final test job does not even try to download the artifacts from build:

Preparing environment
00:00
Running on runner-z4mrjvta-project-110011100-concurrent-0 via ip-10-0-0-111...
Getting source from Git repository
00:01
Skipping Git repository setup
Skipping Git checkout
Skipping Git submodules setup
Executing "step_script" stage of the job script
00:00
$ pwd
/builds/my_project
$ ls -al
total 8
drwxrwxrwx 2 root root 4096 Dec  1 15:31 .
drwxrwxrwx 4 root root 4096 Dec  1 15:31 ..
$ ls -al ../
total 16
drwxrwxrwx 4 root root 4096 Dec  1 15:31 .
drwxrwxrwx 3 root root 4096 Dec  1 15:31 ..
drwxrwxrwx 2 root root 4096 Dec  1 15:31 my_project
drwxrwxrwx 2 root root 4096 Dec  1 15:31 my_project.tmp

Configs

.gitlab-ci.yml

stages:
  - checkout-branch
  - build-branch
  - test-branch

workflow:
  # These rules are global and control whether pipelines are run at all.
  rules:
    # Disable "detatched" pipelines created by "Pipelines for Merge Requests".
    # Works around https://gitlab.com/gitlab-org/gitlab/-/issues/34756
    # Works around https://gitlab.com/gitlab-org/gitlab/-/issues/201845
    - if: '$CI_MERGE_REQUEST_EVENT_TYPE == "detached"'
      when: never
    - if: '$CI_COMMIT_BRANCH'
      when: always
default:
  image: alpine:3.1

checkout-branch:
  stage: checkout-branch
  trigger:
    include:
      - local: ci-config/checkout-branch.yml
    strategy: depend

build-branch: 
  stage: build-branch
  trigger:
    include:
      - local: ci-config/build.yml
    strategy: depend
  needs:
    - job: checkout-branch
      artifacts: true

test-branch:
  stage: test-branch
  trigger:
    include:
      - local: ci-config/test.yml
    strategy: depend
  needs:
    - job: build-branch
      artifacts: true

ci-config/checkout-branch.yml

checkout-branch:
  variables:
    GIT_SUBMODULE_STRATEGY: none
    GIT_STRATEGY: none
    GIT_CHECKOUT: "false"

  image: alpine:3.10

  before_script:
    - 'which git || apk add --no-cache git'
    - git config --global user.password "${CI_JOB_TOKEN}"
    - rm -rf ${CI_PROJECT_NAME}
    - git clone -b "$CI_COMMIT_REF_NAME" https://gitlab-ci-token:${CI_JOB_TOKEN}@gitlab.com/${CI_PROJECT_PATH}.git
    - cd ${CI_PROJECT_NAME}

  script:
    - git submodule foreach git submodule update --init --depth 1

  artifacts:
    paths:
      - "${CI_PROJECT_NAME}"
    expire_in: 1 week

ci-config/build.yml

build: 
  variables:
    GIT_SUBMODULE_STRATEGY: none
    GIT_STRATEGY: none
    GIT_CHECKOUT: "false"

   image: $CI_REGISTRY/our_biz/environments/custom_image:latest

  before_script:
    - echo ${CI_PROJECT_NAME}
    - echo ${CI_COMMIT_REF_SLUG}
    - echo ${CI_COMMIT_SHORT_SHA}

  script:
    - cd ${CI_PROJECT_NAME}
    - dpkg-buildpackage --build=binary

  artifacts:
    name: "my_project_${CI_COMMIT_REF_SLUG}_${CI_COMMIT_SHORT_SHA}"
    paths:
      - "${CI_PROJECT_NAME}/log"
      - "${CI_PROJECT_NAME}/install"
      - "${CI_PROJECT_NAME}/build"
      - "${CI_PROJECT_NAME}/src" 
      - "${CI_PROJECT_NAME}/../${CI_PROJECT_NAME}*.deb"
    expire_in: 1 week

ci-config/test.yml

test:
  tags:
    - vcan
  variables:
    GIT_SUBMODULE_STRATEGY: none
    GIT_STRATEGY: none
    GIT_CHECKOUT: "false"

 image: $CI_REGISTRY/our_biz/environments/custom_image:latest

  before_script:
    - pwd

  script:
    - ls -al
    - ls -al ../

gitlab-runner /etc/gitlab-runner/config.toml

concurrent = 1
check_interval = 10
loglevel = "debug"

[session_server]
  # How long in seconds the session can stay active after the job completes.
  session_timeout = 1800
  # Listen on all available interfaces on port 8093
  listen_address = "0.0.0.0:8093"
  # The URL that the Runner will expose to GitLab to be used to access the session server.
  advertise_address = "vcan-runner.our.biz:8093"


[[runners]]
  name = "docker-runner-vcan"
  url = "https://gitlab.com"
  token = "xxxxxxxxxxxxxx"
  limit = 1
  executor = "docker"
  cache_dir = "/export/gitlab-runner-cache"
  [runners.custom_build_dir]
  [runners.cache]
    [runners.cache.s3]
    [runners.cache.gcs]
    [runners.cache.azure]
  [runners.docker]
    tls_verify = false
    image = "ubuntu:18.04"
    network_mode = "host"
    privileged = false
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = false
    cache_dir = "/export/gitlab-runner-cache"
    volumes = ["/export/gitlab-runner-cache"]
    shm_size = 0

Troubleshooting steps taken

  • Rename job in ci-config/build.yml from build to build-branch
  • Rename build-branch stage job to build
  • Configure build-branch job to capture untracked aritfacts

If anyone could shed any light on why the third job will not download artifacts from the second job it’d be much appreciated.

Thanks for taking the time to read this.