Pipeline Job stopped working - Token not accepted from pipline

GitLab CI/CD
1

TLDR; The problem is that instance CI variables have stopped working. Instead use a project-scoped or a group-scoped CI variable instead.

Problem to solve

What I want to achieve:

  • We have a GitLab instance with multiple projects
  • Some projects depend on releases of other projects
  • In order to access releases from other projects and create releases I created a python package that takes care of that
  • That package uses a personal access token to access the GitLab API
  • We are using this python package from Jenkins as well, everything works there
  • The GitLab pipeline has been working for about 2 years

The problem that now appears:

  • I realized that the pipeline stopped working some weeks ago (probably after some update)
  • It says 401 Unauthorized
  • When the same token is used in Jenkins or from command-line, it works

What does all that mean

I can access the GitLab API from anywhere I want but not from the GitLab pipeline anymore. It has been working but does no longer.

Steps to reproduce

Trigger the pipeline using the python package with the token stored as an instance CI variable GITLAB_TOKEN that is referenced in the pipeline. The job fails. Trigger the same python call from command-line with the same token, it works.

The pipeline script:

variables:
  MY_TOKEN: $GITLAB_ACCESS_TOKEN

image:
  name: "docker-registry.company.local:5000/company/company-build:latest"

stages:
  - build

pages:
  rules:
    - if: $PUBLISH_DOCS == "RUN" || $CI_COMMIT_TAG != null
  stage: build
  before_script:
    - mkdir -p sources/build/_deps
    - >
      python sources/python/gitlab_package.py -u http://git.company.local -t ${MY_TOKEN}
      prepare
      --file sources/dependencies.txt
      --directory sources/build/_deps
    - >
      cmake -G Ninja -B sources/build -DCMAKE_BUILD_TYPE=Release
      -DCMAKE_PREFIX_PATH=${PWD}/sources/build/_deps
      --toolchain ${PWD}/sources/cmake-scripts/ppc32-toolchain.cmake
      -DDOCS_OUTPUT_DIR=${PWD}/public
      sources
    - cmake --build sources/build --target clean
  script:
    - cmake --build sources/build --target docs
  tags:
    - company-build
  artifacts:
    paths:
      - public

Follow-up question

Is there another way to achieve my goal? I don’t care if I cannot use the python package in the GitLab pipeline, as long as I can achieve the same goal otherwise. I am aware of the release-cli but we need more than just creating releases, we also need to be able to access specific versions of releases from dependencies. I am also aware that this python package sort of reinvents the wheel but I could not find a suitable solution that fits our specific needs, why I created that package. Actually the problem is not the python package as it works outside the GitLab pipeline.

Versions

  • [x ] Self-managed
  • GitLab.com SaaS
  • Self-hosted Runners

Versions

  • GitLab v17.1.2
  • GitLab Runner 17.1.0

[EDIT]

I can manually inject the variable when manually triggering the job. This works - so it seems that the instance CI variable is not injected in my pipeline.

I already restarted the instance, no luck. Why is the instance CI variable not injected into my pipeline job?

The variable:

  • Is set at the instance level
  • Is not marked as protected - so it should be available to all jobs, as I understand
  • Has the correct access token stored
  • Is masked
  • (Is correctly named, so no typo or whatsoever)

[EDIT 2]
It turns out that the CI Variable defined at project scope is injected correctly, but the variable at instance scope is not. I’ll try to open a bug report.

[EDIT 3]
It turns out its way too complicated to sign up just to create a bug report, so sorry, no bug report.

[EDIT 4]
(I promise this is the last edit / addition)
Two possible workarounds exist:

  • Use a project-specific CI variable
  • Use a group-scoped CI variable

Both work, just the instance CI variables do not work.