Can't login to registry due to "denied: access forbidden"

I’ve got gitlab ce up and running and I am very happy with it.

Now I am trying to set up the CI/CD process to build a docker image, host it in the registry and deploy this image to production.

gitlab runner works fine, but I’m still having trouble to setup the registry.

I am using docker with the gitlab-ce:latest image and traefik as a reverse proxy

The reverse proxy is configured to expose port 5100 to docker.example.com, gitlabs port 80 is exposed to gitlab.example.com, which seems to work fine. The ssl certificates are handled by traefik and delivered from LetsEncrypt

The gitlab.rb:

 registry_external_url 'https://docker.example.com'
 gitlab_rails['registry_enabled'] = true
 gitlab_rails['registry_host'] = "docker.example.com"
 gitlab_rails['registry_api_url'] = "https://docker.example.com"
 gitlab_rails['registry_issuer'] = "gitlab-issuer"
 gitlab_rails['registry_path'] = "/var/opt/gitlab/gitlab-rails/shared/registry"
 
nginx['listen_port'] = 80
   nginx['proxy_set_headers'] = {
  "Host" => "$http_host_with_default",
  "X-Real-IP" => "$remote_addr",
  "X-Forwarded-For" => "$proxy_add_x_forwarded_for",
  "X-Forwarded-Proto" => "https",
  "X-Forwarded-Ssl" => "on",
  "Upgrade" => "$http_upgrade",
  "Connection" => "$connection_upgrade"
 }

 nginx['http2_enabled'] = false

 registry_nginx['enable'] = true

 registry_nginx['proxy_set_headers'] = {
  "Host" => "$http_host",
  "X-Real-IP" => "$remote_addr",
  "X-Forwarded-For" => "$proxy_add_x_forwarded_for",
  "X-Forwarded-Proto" => "https",
  "X-Forwarded-Ssl" => "on"
 }

 registry_nginx['listen_port'] = 5100
 registry_nginx['listen_https'] = false

When I try to login to the registry with

docker login -u [username] -p [password or generated private token]

All I get is “denied: access forbidden”

For the same reason the job configured from the docker-template with

docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" $CI_REGISTRY

fails with exit code 1

Something from the logs that might help:

gitlab           | Started GET "/jwt/auth?account=[myuser]&client_id=docker&offline_token=[FILTERED]&service=container_registry" for XX.XXX.XXX.XX at 2022-01-10 21:51:40 +0000
gitlab           | Processing by JwtController#auth as HTML
gitlab           |   Parameters: {"account"=>"[myuser]", "client_id"=>"docker", "offline_token"=>"[FILTERED]", "service"=>"container_registry"}
gitlab           | Completed 403 Forbidden in 12ms (Views: 0.3ms | ActiveRecord: 0.0ms | Elasticsearch: 0.0ms | Allocations: 3719)

I am sure, that this is just a small configuration issue which might be pretty easy to fix, but I can’t find the solution by myself.

What did I do wrong? I’m not sure, if I delivered all information that might be needed, so please let me know if something is missing.

Thanks in advance!!!

which permissions does the token have?

The personal token? Full access: api, read_api, read_repository, write_repository, read_registry, write_registry.

I’m not 100% why it works, but the simple solution was to specify the general URL of the gitlab installation by configuring the gitlab.rb with

external_url https://gitlab.example.com

oh, so you access the docker registry directly over the gitlab’s “root” fqdn? And it works just fine?

Docker’s http client is very dumb and while resolving an auth from http://gitlab.example.com/jwt/auth (gitlab behind reverse proxy) docker ignoring redirect to https. I found “exploit” like solution here

OMG, thanks a lot!!! I was banging my head over this problem for a long time and this was very helpful!