We host our own group runners for GitLab on our build server. As of this morning, they’ve stopped being able to authenticate. They’ve begun to exit with the following error:
WARNING: Failed to pull image with policy "always": Error response from daemon: Head "https://registry.gitlab.com/v2/<REMOVED>/manifests/latest": unauthorized: HTTP Basic: Access denied. The provided password or token is incorrect or your account has 2FA enabled and you must use a personal access token instead of a password. See https://gitlab.com/help/user/profile/account/two_factor_authentication#troubleshooting
We have not changed our 2FA requirements recently and haven’t just exited a 2FA grace period. Renewing the runner authentication tokens via the gitlab-runner CLI has not fixed the issue. Un-registering and re-registering the runner has not helped. Disabling 2FA requirements at the group level has not helped.
gitlab-runner verify shows the runners as valid.
We are using GitLab.com SaaS, runner version on the build server is latest from packags, 16.9.1.
The log info you posted, seems to suggest wherever you are pulling the docker image from needs authentication. Perhaps the username that is used during this step has an expired personal access token, or project token has expired. Gitlab a while back forced tokens to expire after 1 year so if you were previously using tokens with no expiry date, they were forced to have one.
It would suggest taking a look at the appropriate .gitlab-ci.yml to find out what step it’s failing at and then checking the username/token in the project/group variables that are doing the login to download the docker image.
1 Like
I hadn’t even thought to check the auth used in the pipeline file. I’d just been blindly assuming the runner itself was providing the auth. This is very likely the issue. Will check and report back.
We’re seeing the same issue on docker login.
Error response from daemon: Get "https://registry.gitlab.com/v2/": unauthorized: HTTP Basic: Access denied. The provided password or token is incorrect or your account has 2FA enabled and you must use a personal access token instead of a password. See https://gitlab.com/help/user/profile/account/two_factor_authentication#troubleshooting
This is CI that we run very frequently without issue. The last time we saw this there was an issue on GitLab’s side.
1 Like
Hi @garethsaul @wes.tibbitts,
As @iwalker mentioned, this may be related your access token. I would recommend rotating your access token. You can review our documentation on Expired Access Tokens to do so.
GitLab removed support for non-expiring access tokens as part of a security best practice to ensure that leaked tokens are not usable. While we did send several notifications via blog and email beginning in September 2022, we recognize that not every customer saw these notifications, and some have experienced a service disruption as a result.
I’ve checked the .gitlab-ci.yml, there are no distinct credentials there. It’s the most basic CI pipeline.
stages:
- test
- deploy
test:
stage: test
image: registry.gitlab.com/<company>/<project>/<dev build container>
script:
- go version
- make test
The authentication issue seems to be pulling the images from the registry. Which credentials are used to do this? The documentation says that the runner uses a “runner authentication token” which it exchanges for a “job token”. Checking the config.toml for the runner, the runner token is fresh (obtained 2024-05-14).
Documentation on CI/CD job tokens says “A job token can access a project’s resources without any configuration, but it might give extra permissions that aren’t necessary”. The registry here is on the same project.
I was able to resolve this issue by configuring docker credentials for the runner as detailed here.
It is still not clear why this was necessary and why this used to work without requiring that step. This is not a private container registry, it is the default container registry for the project.
I cannot find any existing token that might have expired. I have checked:
- runner config.yml
- gitlab-runner user environment variables
- gitlab-runner .docker config
- CI/CD variables for the project
- CI/CD variables for the group
Is the project private though? Or is it a public project that anyone can access without logging into Gitlab?
Usually on mine if my project is private I use CI/CD variables, and reference them in the .gitlab-ci.yml pipeline so that it can login. I suppose if you do it at the runner level, then it’s a single login that requires all access to all projects to work. Whereas using CI/CD variables, I can use different tokens to restrict access to projects so a bit more secure than doing at runner level.
Seems strange though that it just stopped working.
@iwalker Yes, private project. I can understand how and why this fix resolves the issue, but not why it was necessary. All the documentation on the job token suggests it should be authenticated to container registry.
I have a feeling Gitlab changed something, since there are a few people over the past 24 hours reporting the same issue.
Eg: GitLab CI Not Functioning Despite Identical Configurations: Seeking Troubleshooting Assistance
Not entirely sure what could have caused this to be honest. Maybe it will become clear shortly from the Gitlab Team.
In case it helps anyone else, we ran into the same issue and found the reason in our case. It turned out that a host that had this issue had separate registry credentials (i.e. separate from anything managed by gitlab-ci) stored for the root-user on the machine.
Basically, someone did a docker login to the registry some time back (for testing), with a separate token. This token was not supposed to be needed on the host anymore since a long time, but it was still stored in /root/.docker/config.json. After this token expired, we starting seeing the same error-message for any job on this host.
Purging the cached/old credentials (i.e., deleting /root/.docker/config.json) immediately resolved the issue.
So it seems that the docker-commands were in fact using those cached credentials to pull from the docker registry, rather than the credentials managed by gitlab CI itself.
4 Likes
Good help. I checked that (specifically including root) carefully. Although it’s always possible I missed it.
Thanks. It was exactly that. I was having a hard time to locate the cause, because the expiration of the old token fell together with the gitlab upgrade to 17.0.
These are very difficult issues to investigate.