Auth error when CI tries to pull self-hosted registry image

Hi folks,

I am running a self-managed gitlab instance on v12.9.5. Up until the other day, it was using the sameersbn/docker-gitlab containers, but I have now shifted it to use the omnibus build (same version).

After we shifted to the new setup, we saw errors when running CI builds that pulled images from our container registry (the one that ships with GitLab). This had always been handled by the credentials passed with the job itself, and not with an explicitly set DOCKER_AUTH_CONFIG.

Example project to illustrate the bug:

stages:
  - build
    
linting:
  stage: build
  image: registry.gitlab.example.com/puppet/validator:puppet6
  script:
    - echo "Did a thing!"
  tags:
    - test-docker

Results:

 Running with gitlab-runner 12.10.0 (c553af1a)
  on gitlab-runner-docker-2034 w4vdWUe2
Preparing the "docker" executor
Using Docker executor with image registry.gitlab.example.com/puppet/validator:puppet6 ...
Pulling docker image registry.gitlab.example.com/puppet/validator:puppet6 ...
ERROR: Preparation failed: Error response from daemon: Get https://registry.gitlab.example.com/v2/puppet/validator/manifests/puppet6: denied: access forbidden (docker.go:198:0s)

We currently have a work-around of setting DOCKER_AUTH_CONFIG using someone’s credentials. Obviously this is not how we want this to work going forward, though.

Things I’ve tried:

  • For this example project, I locked it down to one brand new runner, to make sure interfacing with the old, already-extant runners wasn’t the issue
  • I also triple-checked that we pulled over the appropriate secrets from the docker installation - which according to the docs, I have. I pulled in db_key_base, secret_key_base, and otp_key_base, as well as the gitlab shell secret token.

Any ideas or thoughts on this would be very welcome. Thanks.

Hi @commandertso

what kind of Gitlab Runner executor do you use?

Hey Balonik - thanks for the reply.

I was able to reproduce this using gitlab-runner in a docker container, running as a shell executor. I saw the error on extant runners after the upgrade, and then saw it on a new runner as well.

You can look into logs registry/current or gitlab-rails/production.log to get some hints why the authentication failed.

Sure, thanks.

registry/current:

2022-04-05_17:51:56.57542 time="2022-04-05T13:51:56.57500514-04:00" level=warning msg="error authorizing context: authorization token required" go.version=go1.12.13 http.request.host=registry.gitlab.example.com http.request.id=79c90813-6aca-477a-bc7c-6fb7f7087310 http.request.method=GET http.request.remoteaddr=XXX.XXX.XXX.XXX http.request.uri="/v2/" http.request.useragent="docker/18.09.0 go/go1.10.4 git-commit/4d60db4 kernel/3.10.0-957.1.3.el7.x86_64 os/linux arch/amd64 UpstreamClient(Go-http-client/1.1)"

gitlab-rails/production.log:

Started GET "/<NAMESPACE>/<REPO_NAME>/-/jobs/250907.json" for <IP_ADDRESS> at 2022-04-05 14:24:48 -0400
  Parameters: {"service"=>"git-upload-pack", "namespace_id"=>"common", "repository_id"=>"opennebula-context.git"}
Completed 200 OK in 23ms (Views: 0.4ms | ActiveRecord: 3.2ms | Elasticsearch: 0.0ms | Allocations: 5315)
Started GET "/<NAMESPACE>/<REPO_NAME>/-/jobs/250907/trace.json?state=eyJvZmZzZXQiOjEwOTIsInN0eWxlIjp7ImZnIjpudWxsLCJiZyI6bnVsbCwibWFzayI6MH0sIm9wZW5fc2VjdGlvbnMiOnsicHJlcGFyZS1leGVjdXRvciI6IjE2NDkxODMwNzQifX0%3D" for <IP_ADDRESS> at 2022-04-05 4:24:50 -0400
Processing by Projects::JobsController#trace as JSON
  Parameters: {"state"=>"eyJvZmZzZXQiOjEwOTIsInN0eWxlIjp7ImZnIjpudWxsLCJiZyI6bnVsbCwibWFzayI6MH0sIm9wZW5fc2VjdGlvbnMiOnsicHJlcGFyZS1leGVjdXRvciI6IjE2NDkxODMwNzQifX0=", "namespace_id"=>"<NAMESPACE>", "project_id"=>"<REPO_NAME>", "id"=>"250907"}
Started GET "/<NAMESPACE>/<REPO_NAME>/-/jobs/250907.json" for <IP_ADDRESS> at 2022-04-05 14:24:58 -0400
Processing by Projects::JobsController#show as JSON
  Parameters: {"namespace_id"=>"<NAMESPACE>", "project_id"=>"<REPO_NAME>", "id"=>"250907"}

The “authorization token required” from the registry log makes sense, but doesn’t tell me much more than I already knew, I think.

I didn’t see anything else that stuck out to me - please let me know if you do. Thanks.