Docker registry - authorization token requiered

Hi there,

I’m currently setting up gitlab as our main docker registry, as it was described as “easy” and “straight forward”.
In fact, I’m stuck at logging in to the registry.

Here’s the config:
registry_external_url ‘https://gitlab.mycompany.intern:4567

### Settings used by GitLab application
gitlab_rails['registry_enabled'] = true
gitlab_rails['registry_host'] = "gitlab.mycompany.intern"
gitlab_rails['registry_port'] = "5000"
gitlab_rails['registry_path'] = "/var/opt/gitlab/gitlab-rails/shared/registry"

###! **Do not change the following 3 settings unless you know what you are
###!   doing**
gitlab_rails['registry_api_url'] = "http://localhost:5000/"
gitlab_rails['registry_key_path'] = "/var/opt/gitlab/gitlab-rails/etc/gitlab-registry.key"
gitlab_rails['registry_issuer'] = "container_registry"

### Settings used by Registry application
registry['enable'] = true
registry['username'] = "registry"
registry['group'] = "registry"
registry['dir'] = "/var/opt/gitlab/gitlab-rails/shared/registry"
registry['registry_http_addr'] = "localhost:5000"
registry['log_directory'] = "/var/log/gitlab/registry"
registry['log_level'] = "debug"
registry['rootcertbundle'] = "/var/opt/gitlab/registry/certs/registry-auth.crt"

And when i try to login via
docker login gitlab.mycompany.intern:4567

I get this error:

Username: MYADMINACCOUNT
Password: 
Error response from daemon: login attempt to https://gitlab.mycompany.intern:4567/v2/ failed with status: 401 Unauthorized

Here are the logs:

==> /var/log/gitlab/gitlab-rails/production.log <==
Started POST "/api/v4/jobs/request" for 127.0.0.1 at 2017-05-29 10:19:40 +0000
Started GET "/jwt/auth?account=MYADMINACCOUNT&client_id=docker&offline_token=true&service=container_registry" for 192.168.100.237 at 2017-05-29 10:19:40 +0000
Processing by JwtController#auth as HTML
  Parameters: {"account"=>"MYADMINACCOUNT", "client_id"=>"docker", "offline_token"=>"true", "service"=>"container_registry"}
Filter chain halted as :authenticate_project_or_user rendered or redirected
Completed 401 Unauthorized in 106ms (Views: 0.2ms | ActiveRecord: 3.8ms)

==> /var/log/gitlab/gitlab-workhorse/current <==
2017-05-29_10:19:40.11828 gitlab.mycompany.intern @ - - [2017-05-29 10:19:40.111310696 +0000 UTC] "POST /api/v4/jobs/request HTTP/1.1" 204 0 "" "gitlab-ci-multi-runner 9.2.0 (9-2-stable; go1.7.5; linux/amd64)" 0.006924
2017-05-29_10:19:40.89268 gitlab.mycompany.intern @ - - [2017-05-29 10:19:40.78153977 +0000 UTC] "GET /jwt/auth?account=MYADMINACCOUNT&client_id=docker&offline_token=true&service=container_registry HTTP/1.1" 401 74 "" "docker/17.03.1-ce go/go1.7.5 git-commit/c6d412e kernel/4.4.0-78-generic os/linux arch/amd64 UpstreamClient(Docker-Client/17.03.1-ce \\(linux\\))" 0.111097

==> /var/log/gitlab/nginx/gitlab_access.log <==
127.0.0.1 - - [29/May/2017:10:19:40 +0000] "POST /api/v4/jobs/request HTTP/1.1" 204 0 "-" "gitlab-ci-multi-runner 9.2.0 (9-2-stable; go1.7.5; linux/amd64)"
192.168.100.237 - MYADMINACCOUNT [29/May/2017:10:19:40 +0000] "GET /jwt/auth?account=MYADMINACCOUNT&client_id=docker&offline_token=true&service=container_registry HTTP/1.1" 401 74 "-" "docker/17.03.1-ce go/go1.7.5 git-commit/c6d412e kernel/4.4.0-78-generic os/linux arch/amd64 UpstreamClient(Docker-Client/17.03.1-ce \x5C(linux\x5C))"

==> /var/log/gitlab/nginx/gitlab_registry_access.log <==
192.168.100.237 - - [29/May/2017:10:19:40 +0000] "GET /v2/ HTTP/1.1" 401 87 "-" "docker/17.03.1-ce go/go1.7.5 git-commit/c6d412e kernel/4.4.0-78-generic os/linux arch/amd64 UpstreamClient(Docker-Client/17.03.1-ce \x5C(linux\x5C))"

==> /var/log/gitlab/registry/current <==
2017-05-29_10:19:40.76495 time="2017-05-29T10:19:40.764889919Z" level=debug msg="authorizing request" environment=production go.version=go1.8.1 http.request.host="gitlab.mycompany.intern:4567" http.request.id=1b3a40ba-0077-4918-bd9f-ccffd7f0f735 http.request.method=GET http.request.remoteaddr=192.168.100.237 http.request.uri="/v2/" http.request.useragent="docker/17.03.1-ce go/go1.7.5 git-commit/c6d412e kernel/4.4.0-78-generic os/linux arch/amd64 UpstreamClient(Docker-Client/17.03.1-ce \\(linux\\))" instance.id=04493522-1bea-4153-8616-2a3b258e3efc service=registry version=v2.6.1-1-gdd544a8 
2017-05-29_10:19:40.76504 time="2017-05-29T10:19:40.765009263Z" level=warning msg="error authorizing context: authorization token required" environment=production go.version=go1.8.1 http.request.host="gitlab.mycompany.intern:4567" http.request.id=1b3a40ba-0077-4918-bd9f-ccffd7f0f735 http.request.method=GET http.request.remoteaddr=192.168.100.237 http.request.uri="/v2/" http.request.useragent="docker/17.03.1-ce go/go1.7.5 git-commit/c6d412e kernel/4.4.0-78-generic os/linux arch/amd64 UpstreamClient(Docker-Client/17.03.1-ce \\(linux\\))" instance.id=04493522-1bea-4153-8616-2a3b258e3efc service=registry version=v2.6.1-1-gdd544a8 
2017-05-29_10:19:40.76507 127.0.0.1 - - [29/May/2017:10:19:40 +0000] "GET /v2/ HTTP/1.0" 401 87 "" "docker/17.03.1-ce go/go1.7.5 git-commit/c6d412e kernel/4.4.0-78-generic os/linux arch/amd64 UpstreamClient(Docker-Client/17.03.1-ce \\(linux\\))"

Any ideas how to track down this issue? Or any ideas on how to fix it?
It is currently a show-stopping issue and prevents us from going on with gitlab as our primary dev and CI tool.

Thanks all!

OS:
Distributor ID: Ubuntu
Description: Ubuntu 16.04.2 LTS
Release: 16.04
Codename: xenial
GitLab:
ii gitlab-ce 9.2.2-ce.0 amd64 GitLab Community Edition (including NGINX, Postgres, Redis)

As it seems impossible to pass self-signed certificates to the gitlab-registry process, I switched over to use an external registry (also self-signed certificates).

Now I can login to the registry via docker login but I can’t use gitlab’s registry tab.
When I navigate to the repository and click the “registry” tab all I get is this nice 500 error page.

The logs are not really helpful at all:

Started GET "/MYADMINACCOUNT/test/container_registry" for 192.168.100.237 at 2017-05-30 13:52:44 +0000
Processing by Projects::Registry::RepositoriesController#index as HTML
  Parameters: {"namespace_id"=>"MYADMINACCOUNT", "project_id"=>"test"}
Completed 500 Internal Server Error in 175ms (ActiveRecord: 18.3ms)

Faraday::ConnectionFailed (end of file reached):
  



==> /var/log/gitlab/gitlab-workhorse/current <==
2017-05-30_13:52:45.42309 2017/05/30 13:52:45 ErrorPage: serving predefined error page: 500
2017-05-30_13:52:45.42328 gitlab.mycompany.intern @ - - [2017-05-30 13:52:44.796467134 +0000 UTC] "GET /MYADMINACCOUNT/test/container_registry HTTP/1.1" 500 2911 "" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/58.0.3029.110 Chrome/58.0.3029.110 Safari/537.36" 0.626700

the gitlab.rb:

gitlab_rails['registry_enabled'] = true
gitlab_rails['registry_host'] = "gitlab.mycompany.intern"
gitlab_rails['registry_path'] = "/var/opt/gitlab/gitlab-rails/shared/registry"

####! **Do not change the following 3 settings unless you know what you are
####!   doing**
gitlab_rails['registry_api_url'] = "http://localhost:5000/"
gitlab_rails['registry_key_path'] = "/data/docker/registry/certs/registry-auth.key"
gitlab_rails['registry_issuer'] = "gitlab-issuer"

The registry is started via:

docker run -d -p 5000:5000 --restart=always --net dockernet --name registry -v /data/docker/registry/config/config.yml:/etc/docker/registry/config.yml -v /data/docker/registry/certs/auth.crt:/root/certs/auth.crt -v /etc/ssl:/etc/ssl -v /var/opt/gitlab/gitlab-rails/shared/registry/:/var/lib/registry registry

And the registry’s config:

version: 0.1
log:
  accesslog:
    disabled: false
  level: debug
  formatter: text
  fields:
    service: registry
    environment: production
storage:
  filesystem:
    rootdirectory: /var/lib/registry
    maxthreads: 100
  delete:
    enabled: false
  redirect:
    disable: false
  cache:
    blobdescriptor: redis
  maintenance:
    uploadpurging:
      enabled: true
      age: 168h
      interval: 24h
      dryrun: false
    readonly:
      enabled: false
auth:
  token:
    realm: https://gitlab.mycompany.intern/jwt/auth
    service: container_registry
    issuer: gitlab-issuer
    rootcertbundle: /root/certs/auth.crt
http:
  addr: localhost:5000
  secret: totalrandomsecureTocken
  relativeurls: false
  debug:
    addr: localhost:5001
  headers:
    X-Content-Type-Options: [nosniff]
  http2:
    disabled: false
redis:
  addr: localhost:6379
  password: asecret
  db: 0
  dialtimeout: 10ms
  readtimeout: 10ms
  writetimeout: 10ms
  pool:
    maxidle: 16
    maxactive: 64
    idletimeout: 300s
validation:
  enabled: false
  manifests:
    urls:
      allow:
        - ^https?://([^/]+\.)*example\.com/
      deny:
        - ^https?://www\.example\.com/

I didn’t change much from the default config, just changed the hostnames and paths to match gitlab.rb.

Any ideas?

Thanks and regards,
Stephan