Using CI_JOB_TOKEN with GitLab API to access artifacts from another private repository

GitLab CI/CD
,
1

Problem to solve

Inside a private group (created by me) there are two private repositories (also created by me) - SOURCE and TARGET. When executing a GitLab Job Artifacts API call to retrieve latest artifacts from SOURCE in a CI job that is part of TARGET error 404 Not Found is returned.

Steps to reproduce

  • Create two private repositories SOURCE and TARGET
  • Create a CI job in SOURCE that produces an artifact
  • Add TARGET project to Allow access to of SOURCE
  • Create a CI job in TARGET that uses CI_JOB_TOKEN in a GET request (set header to include JOB-TOKEN with the value of the CI_JOB_TOKEN) to retrieve the latest successfully build artifacts from SOURCE
  • Trigger pipeline in SOURCE that successfully completes and produces the respective artifact
  • Trigger pipeline in TARGET

Configuration

I use the following CI job to retrieve the artifacts using CMake’s FetchContent module:

.gitlab-ci.yml (only job that is doing the API call to retrieve the artifacts

build-demo:
  stage: build
  rules:
    - if: $CI_COMMIT_REF_NAME == "main"
      allow_failure: true
  artifacts:
    untracked: true
    paths:
      - build
      - cmake
      - token.txt
    when: always
  before_script:
    - |
      $CI_JOB_TOKEN | Out-File token.txt -NoNewline
      (Get-Content ./cmake/download.cmake).Replace('PRIVATE-TOKEN', 'JOB-TOKEN') | Set-Content ./cmake/download.cmake
  script:
    - cmake -Bbuild -G "Visual Studio 16 2019" -S. -DCMAKE_CONFIGURATION_TYPES="release"
    - cmake --build build --target ALL_BUILD --config release

download.cmake

function(download_file_gitlab_latest name token proj_id branch_tag job_name)
    set(_URL "https://gitlab.example.com/api/v4/projects/${proj_id}/jobs/artifacts/${branch_tag}/download?job=${job_name}")
    set(_HEADER "PRIVATE-TOKEN: ${token}")
    message("GET Request: ${_URL}")
    message("Destination prefix: ${name}")

    FetchContent_Declare(
        ${name}
        URL "${_URL}"
        HTTP_HEADER ${_HEADER}
        DOWNLOAD_NO_EXTRACT 0
        DOWNLOAD_EXTRACT_TIMESTAMP 1
        DOWNLOAD_NO_PROGRESS 0
    )

    if(NOT ${name}_POPULATED)
        FetchContent_MakeAvailable(${name})
    endif()
endfunction(download_file_gitlab_latest)

CMakeLists.txt (only section where download is triggered)

file(
    STRINGS
    "${CMAKE_SOURCE_DIR}/token.txt"
    TOKEN
    LIMIT_COUNT 1
)

download_file_gitlab_latest(
                "downloaded_deps"
                ${TOKEN}
                68522
                "${config}"
                "build-all-${config}"
            )

An example for a call contains the following URL:

http://gitlab.example.com/api/v4/projects/68522/jobs/artifacts/release/download?job=build-all-release

Versions

  • Self-managed, Premium features
  • Self-hosted Runners

Versions of both GitLab instance and runner are latest stable.

Notes

I read that a Premium feature is needs: project may solve my problem. However, the documentation on that is sparse at best and it also would mean that I will have to adapt two workflows for managing the downloading of artifacts based on whether I do that locally or within a CI job. I prefer to just generate my token, dump it into a file and load it in my CMake project during configuration plus patch the header’s parameter.

If I use a PRIVATE-TOKEN in the header with a value that comes from a Group Access Token or Personal Access Token I don’t have any issues, meaning that it’s the CI_JOB_TOKEN that creates a problem. Running the configuration step that triggers the API call locally (using a PAT) yields the expected results (artifacts are downloaded).

2

I’m not sure what

Add TARGET project to Allow access to of SOURCE

implies, but my understanding is that access granted via CI_JOB_TOKEN is limited to the project that runs the job, i.e. TARGET in your case.

As you mentioned, using a Group Access Token (if both project share a common group) or a Personal Access Token (if that user is a sufficiently privileged member of both projects) will work. Have a look at Project Access Tokens if you want to limit the scope of the token further.