Deploy token login fails for registry

I have just migrated my GitLab installation from Omnibus to Docker all-in-one. When I try to docker login to the internal registry, the login fails with denied: access forbidden. I tried that using a group deploy token and a personal access token, both with read_registry permission. It works in CI with the dynamic token.

In the logs, it seems that the /jwt/auth-Controller always returns 403.

==> /var/log/gitlab/nginx/gitlab_registry_access.log <==
172.19.0.1 - - [11/Nov/2020:08:57:54 +0000] "GET /v2/ HTTP/1.1" 401 87 "" "docker/19.03.13 go/go1.13.15 git-commit/4484c46d9d kernel/4.19.0-12-amd64 os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.13 \x5C(linux\x5C))" -

==> /var/log/gitlab/gitlab-rails/production.log <==
Started GET "/jwt/auth?account=t-deploy&client_id=docker&offline_token=[FILTERED]&service=container_registry" for 127.0.0.1 at 2020-11-11 08:57:54 +0000
Processing by JwtController#auth as HTML
  Parameters: {"account"=>"t-deploy", "client_id"=>"docker", "offline_token"=>"[FILTERED]", "service"=>"container_registry"}
Completed 403 Forbidden in 9ms (Views: 0.2ms | ActiveRecord: 0.0ms | Elasticsearch: 0.0ms | Allocations: 5040)

==> /var/log/gitlab/gitlab-rails/production_json.log <==
{"method":"GET","path":"/jwt/auth","format":"html","controller":"JwtController","action":"auth","status":403,"time":"2020-11-11T08:57:54.311Z","params":[{"key":"account","value":"t-deploy"},{"key":"client_id","value":"docker"},{"key":"offline_token","value":"[FILTERED]"},{"key":"service","value":"container_registry"}],"remote_ip":"127.0.0.1","user_id":null,"username":null,"ua":"docker/19.03.13 go/go1.13.15 git-commit/4484c46d9d kernel/4.19.0-12-amd64 os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.13 \\(linux\\))","correlation_id":"gKSW0Nvd2I1","meta.caller_id":"JwtController#auth","meta.feature_category":"authentication_and_authorization","redis_calls":7,"redis_duration_s":0.001061,"redis_read_bytes":1368,"redis_write_bytes":412,"redis_cache_calls":7,"redis_cache_duration_s":0.001061,"redis_cache_read_bytes":1368,"redis_cache_write_bytes":412,"queue_duration_s":0.002285,"cpu_s":0.01,"db_duration_s":0.0,"view_duration_s":0.00016,"duration_s":0.00892,"db_count":0,"db_write_count":0,"db_cached_count":0}

I have already run the gitlab self check and it looks all green.

How do I make the registry work with access tokens again?

This is due to a bug in the omnibus setup.

My setup consists of a reverse proxy that terminates TLS for the web interface but not the registry which handles TLS for itself.
Certificates are passed through from the reverse proxy as a docker mount.

Because of this setup, the cookbook will generate a http:// token realm in the registry config.yml when it should really be https://. Changing the line and restarting the registry fixes the problem until the next run of gitlab-ctl reconfigure.

Now I will need to find a permanent solution.

1 Like

Hi @BlackCetha,

I was having the same problem as you have described here, and indeed making the change in the registry config.yml has fixed the problem. Thanks a lot for that.

Now I am curious if you have managed to find a permanent solution to this? Since this is what I will start hunting for next :slight_smile:

Had the exact same problem, this fixed it (editing the registry’s config.yml). Reference: gitlab docker container v14.5.0-ce.0