Gitlab Docker Registry integration: "token signed by untrusted key" error

How to Use GitLab
1

Hi there,

Today I tried to integrate the Gitlab Registry feature with our already existing Docker Registry which successfully uses SAML authentication via Keycloak - unfortunately without any success.

When I try to load the “Registry” menu of a testing repository, the following error is shown in the Docker registry log:

time="2018-04-09T16:20:16Z" level=info msg="token signed by untrusted key with ID: \"BA4N:DCLK:FGIV:KNQT:3XEP:B453:7VKX:C6LV:Q6ZD:AMTY:GUT4:UUKZ\""
time="2018-04-09T16:20:16Z" level=warning msg="error authorizing context: invalid token" go.version=go1.7.6 http.request.host=registry.example.com http.request.id=6bbb5649-6fe6-425b-93de-3a4cd958c322 http.request.method=GET http.request.remoteaddr=1.1.1.1 http.request.uri="/v2/testers/testing-demo-repo/tags/list" http.request.useragent="Faraday v0.12.2" instance.id=b2c491bd-1a60-4df4-8843-4704c7a2dfe3 vars.name="testers/testing-demo-repo" version=v2.6.2
172.22.0.1 - - [09/Apr/2018:16:20:16 +0000] "GET /v2/testers/testing-demo-repo/tags/list HTTP/2.0" 401 168 "" "Faraday v0.12.2"
time="2018-04-09T16:20:17Z" level=info msg="token signed by untrusted key with ID: \"BA4N:DCLK:FGIV:KNQT:3XEP:B453:7VKX:C6LV:Q6ZD:AMTY:GUT4:UUKZ\""
time="2018-04-09T16:20:17Z" level=warning msg="error authorizing context: invalid token" go.version=go1.7.6 http.request.host=registry.example.com http.request.id=6214c5c4-e901-4eda-9885-f3f727d8d6f1 http.request.method=GET http.request.remoteaddr=1.1.1.1 http.request.uri="/v2/testers/testing-demo-repo/tags/list" http.request.useragent="Faraday v0.12.2" instance.id=b2c491bd-1a60-4df4-8843-4704c7a2dfe3 vars.name="testers/testing-demo-repo" version=v2.6.2
172.22.0.1 - - [09/Apr/2018:16:20:17 +0000] "GET /v2/testers/testing-demo-repo/tags/list HTTP/2.0" 401 168 "" "Faraday v0.12.2"

Does anyone know where this come from? The Registry configuration looks like the following:

registry:
  image: registry:2.6.2
  volumes:
    - /opt/registry/certs:/opt/certs/:Z
    - /opt/registry/data:/opt/data:Z
  environment:
    REGISTRY_STORAGE_FILESYSTEM_ROOTDIRECTORY: /opt/data
    REGISTRY_HTTP_TLS_CERTIFICATE: /opt/certs/localhost.crt
    REGISTRY_HTTP_TLS_KEY: /opt/certs/localhost.key
    REGISTRY_AUTH_TOKEN_REALM: https://auth.example.com/auth/realms/myrealm/protocol/docker-v2/auth
    REGISTRY_AUTH_TOKEN_SERVICE: container_registry
    REGISTRY_AUTH_TOKEN_ISSUER: https://auth.example.com/auth/realms/myrealm
    REGISTRY_AUTH_TOKEN_ROOTCERTBUNDLE: /opt/certs/localhost_trust_chain.pem
    REGISTRY_HTTP_SECRET: <secret-string>
  labels:
    traefik.port: 5000
    traefik.frontend.rule: Host:registry.example.com
    traefik.protocol: https

And the Gitlab configuration like this:

globalgit:
  image: gitlab/gitlab-ce:10.6.3-ce.0
  environment:
    GITLAB_OMNIBUS_CONFIG: |
      ### General GitLab settings
      external_url = 'https://gitlab.example.com'
      nginx['listen_port'] = 80
      nginx['listen_https'] = false

      ### Registry settings
      # Settings used by GitLab application
      registry['enable'] = false
      registry_external_url = 'https://registry.example.com'
      registry['registry_http_addr'] = "registry.example.com:443"
      gitlab_rails['registry_enabled'] = true
      gitlab_rails['registry_host'] = "registry.example.com"
      #gitlab_rails['registry_port'] = "443"
      gitlab_rails['registry_issuer'] = "https://auth.example.com/auth/realms/myrealm"
      gitlab_rails['registry_api_url'] = "https://registry.example.com"
      gitlab_rails['registry_key_path'] = "/var/opt/gitlab/gitlab-rails/etc/registry.key"
      gitlab_rails['gitlab_default_projects_features_container_registry'] = true
      registry['internal_key'] = "-----BEGIN PRIVATE KEY-----\nmy_secret_private_key\n-----END PRIVATE KEY-----"

  volumes:
    - /opt/gitlab/config:/etc/gitlab:Z
    - /opt/gitlab/logs:/var/log/gitlab:Z
    - /opt/gitlab/data:/var/opt/gitlab:Z
  labels:
    - traefik.port=80
    - traefik.frontend.rule=Host:gitlab.example.com
    - traefik.protocol=http

I already checked, if the private and public key belong to each other what they do:

node1:certs phil$ openssl pkey -in localhost.key -pubout -outform pem | openssl md5
(stdin)= 1b898caff9d7868e9821610e22f5acb0
node1:certs phil$ openssl x509 -in localhost.crt -pubkey -noout -outform pem | openssl md5
(stdin)= 1b898caff9d7868e9821610e22f5acb0

Based on issues/25967 I also ready checked the key ID. Gitlab’s uses the key ID BA4N:DCLK:FGIV:KNQT:3XEP:B453:7VKX:C6LV:Q6ZD:AMTY:GUT4:UUKZ (Base32). In hex thats 0838D1896A2991553613DDC8F0F3BBFD5571797587B23032783527CA5159. The public key part from the localhost.key is the same except the last 4 digits:

node1:certs phil$ openssl dgst -sha256 localhost.pub.der
SHA256(localhost.pub.der)= 0838d1896a2991553613ddc8f0f3bbfd5571797587b23032783527ca5159a318

Does anyone know if that could be the issue? And if so, whats the problem here? Any hints are welcome!

Thanks!

Regards,
Philip

2

FIXED!
I had the same problem on Omnibus.
I figured out that the Certificate and Key on the “registry” part of /etc/gitlab/gitlab-secrets.json MUST MATCH with the ones in following configurations on /etc/gitlab/gitlab.rb :

registry['rootcertbundle'] 
gitlab_rails['registry_key_path']

Now, to be sure that the certificate matches the key, you can use the openssl command:

 openssl x509 -noout -modulus -in server.crt | openssl md5
 openssl rsa -noout -modulus -in server.key | openssl md5
1 Like
3

It seems to be ok to copy & paste the “internal_certificate” from gitlab to the registry and inject it as the REGISTRY_AUTH_TOKEN_ROOTCERTBUNDLE

4

hi, i still confuse to connecting my self-hosted gitlab with self-hosted external registry.
I use docker-v2 auth via keycloak, export key, crt, and pem file from it.
Then on my registry config, i set it to use auth token, with rootcertbundle is pem file
everything seems fine because now i have to use docker login first to pull/push image.

than in gitlab i configure my gitlab, but it wont to read the container registry on each project

if i dont give the registry an auth, it works, the container show in gitlab project’s list

the part that i guess give an error is this line
gitlab_rails[‘registry_key_path’] = “/var/opt/gitlab/gitlab-rails/etc/registry.key”

is registry.key same with localhost.key that i got from keycloak? or should i create another private key?

thanks