CI authentication with CI_REGISTRY_PASSWORD still fails after reverting require_admin_two_factor_authentication to false

GitLab CI/CD
, ,
1

Problem to solve

After enabling require_admin_two_factor_authentication the usage of CI_REGISTRY_PASSWORD fails with the following error:

Pulling docker image <self-hosted-gitlab-domain>:22443/my-group/my-project:my-tag ...
WARNING: Failed to pull image with policy "always": Error response from daemon: Head "https://<self-hosted-gitlab-domain>:22443/v2/my-group/my-project/manifests/my-tag": unauthorized: HTTP Basic: Access denied. The provided password or token is incorrect or your account has 2FA enabled and you must use a personal access token instead of a password. See https://<self-hosted-gitlab-domain>/help/user/profile/account/two_factor_authentication#troubleshooting (manager.go:250:0s)

Based on the information provided in Troubleshooting two-factor authentication | GitLab we could use a token instead. This indeed would solve the problem, e.g. by using CI_JOB_TOKEN.

However, as we want do not want to break all .gitlab-ci.yml immediately, we rather consider reverting require_admin_two_factor_authentication to false, use some time to enforce all projects to swap to CI_JOB_TOKEN and only then enable require_admin_two_factor_authentication.

But THAT seems not to work. Even when require_admin_two_factor_authentication is reverted to false, above error message remains, while the exact same configuration worked before toggling the value of require_admin_two_factor_authentication from false to true and back to false.

Steps to reproduce

On self-hosted GitLab instance, login as administrator and

  1. enable 2FA for admin users. Save.
  2. disable 2FA for admin users. Save.

Configuration

Works:

docker-login:  
  image:
    name: docker:25.0-git
    pull_policy: always
  script:
    - echo "${CI_JOB_TOKEN}" | docker login -u "${CI_REGISTRY_USER}" --password-stdin  "${CI_REGISTRY}"

Fails:

docker-login:  
  image:
    name: docker:25.0-git
    pull_policy: always
  script:
    - echo "${CI_REGISTRY_PASSWORD}" | docker login -u "${CI_REGISTRY_USER}" --password-stdin  "${CI_REGISTRY}"

Versions

  • Self-managed
  • GitLab.com SaaS
  • Dedicated
  • Self-hosted Runners
$ gitlab-rake gitlab:env:info
System information
System:
Current User:   git
Using RVM:      no
Ruby Version:   3.1.5p253
Gem Version:    3.5.11
Bundler Version:2.5.11
Rake Version:   13.0.6
Redis Version:  7.0.15
Sidekiq Version:7.1.6
Go Version:     unknown

GitLab information
Version:        17.2.7
Revision:       41fa57bb8d5
Directory:      /opt/gitlab/embedded/service/gitlab-rails
DB Adapter:     PostgreSQL
DB Version:     14.11
URL:            https://git.dev.leitwert.ch
HTTP Clone URL: https://git.dev.leitwert.ch/some-group/some-project.git
SSH Clone URL:  ssh://git@git.dev.leitwert.ch:1022/some-group/some-project.git
Using LDAP:     no
Using Omniauth: yes
Omniauth Providers:

GitLab Shell
Version:        14.37.0
Repository storages:
- default:      unix:/var/opt/gitlab/gitaly/gitaly.socket
GitLab Shell path:              /opt/gitlab/embedded/service/gitlab-shell

Gitaly
- default Address:      unix:/var/opt/gitlab/gitaly/gitaly.socket
- default Version:      17.2.7
- default Git Version:  2.45.2

$ gitlab-runner -version
Version:      17.2.1
Git revision: 9882d9c7
Git branch:   17-2-stable
GO version:   go1.22.5
Built:        2024-07-25T17:34:53+0000
OS/Arch:      linux/amd64
2

I managed to fix the problem. The hint regarding the root cause was actually in the job log some lines earlier:

Preparing the "docker" executor 00:03
Using Docker executor with image <self-hosted-gitlab-domain>:22443/my-group/my-project:my-tag ...
Authenticating with credentials from /root/.docker/config.json

The root/docker/config.json was configured with some old authentication which seems to have been invalidated by the actions described.

Solution to the problem was to remove the /root/.docker/config.json file.