Container Registry can't be reacher, but shows in WebUI

Hi all,

Just installed GitLab, as I’d like to move away from hosting on GitHub and DockerHub. Loving it so far, and got all my repos pulled in perfectly, worked super easily.

Struggling a bit with the built in container registry however, as I can’t see to connect to it either locally or remotely.

Setup:

User --> Cloudflare --> Traefik Reverse Proxy --> Dedicated VM running GitLab Omnibus

Debian 10 VM (dedicated) that's only available on a local IP
GitLab omnibus installed directly (not containerised) 

Running the following command:

docker login -u user registry.example.com

Brings a certificate error:

Error response from daemon: Get https://registry.example.com/v2/: Get https://192.168.11.28/jwt/auth?account=tom&client_id=docker&offline_token=true&service=container_registry: x509: cannot validate certificate for 192.168.11.28 because it doesn't contain any IP SANs

registry settings in gitlab.rb:

registry_external_url 'https://registry.example.com:5050'

### Settings used by GitLab application
gitlab_rails['registry_enabled'] = true
gitlab_rails['registry_host'] = "registry.example.com"
gitlab_rails['registry_port'] = "5000"
gitlab_rails['registry_path'] = "/var/opt/gitlab/gitlab-rails/shared/registry"

registry_nginx settings in gitlab.rb:

registry_nginx['enable'] = true

registry_nginx['proxy_set_headers'] = {
 "Host" => "$http_host",
 "X-Real-IP" => "$remote_addr",
 "X-Forwarded-For" => "$proxy_add_x_forwarded_for",
 "X-Forwarded-Proto" => "https",
 "X-Forwarded-Ssl" => "on"
}

# When the registry is automatically enabled using the same domain as `external_url`,
# it listens on this port
registry_nginx['listen_port'] = 5050
registry_nginx['ssl_certificate'] = "/etc/gitlab/ssl/cert.name.doesn't.match.URL.pem"
registry_nginx['ssl_certificate_key'] = "/etc/gitlab/ssl/key.name.doesn't.match.URL.key"

Any help would be greatly appreciated as I’ve searched and searched, messed around with loads of stuff and I’m completely stumped at this point.

Hi,

Thrown this into Google because I have never seen this specific TLS error with Docker. Brings up Docker machine can't validate certificate because it doesn't contain any IP SANs · Issue #4369 · docker/machine · GitHub pointing to a proxy issue. Since you mentioned Traefik as proxy, adding this to the search landing at x509: cannot validate certificate for 192.168.16.7 because it doesn't contain any IP SANs · Issue #3906 · traefik/traefik · GitHub

It seems to be related to how Traefik terminates the TLS endpoint, or, routes the traffic from the external registry.gitlab.com to the internal IP address and port 5050.

Can't access docker registry behind traefik 2.0 (#5560) · Issues · GitLab.org / omnibus-gitlab · GitLab might give a few pointers too. Can you try the suggested registry_http_addr key, adjusted to port 5050?

registry['registry_http_addr'] = "0.0.0.0:5050"

Note: I’m guessing here, I have never setup this up yet.

Cheers,
Michael

Thanks, I hadn’t found that one relating to traefik – I was under the impression those settings were for a non-gitlab registry using GitLab as auth?

Will give it a go!

True, now that you mention own registry, this is the wrong path forward. Sorry, please ignore that idea.

Another idea:

Traefik routes the entire traffic from https://registry.example.com:5050 to internal.ip.on.the.vm:5050.

The SNI (Server Name Identification) error means that the resolved FQDN from the TLS certificate does not match the IP address, and it bails with a TLS error in the client-server handshake.

Can you share the Traefik configuration, specifically the rules? I can see these settings as docker-compose labels in blog posts like these: https://community.hetzner.com/tutorials/gitlab-server-with-docker - might be an idea to verify they are correct.

Cheers,
Michael

Sure, no problem.

http:
  routers:
    gitlab-registry-all:
      entryPoints:
        - https
      rule: Host(`registry.example.com`)
      middlewares:
        - chain-cfp-public@file
      service: gitlab-registry-svc

  services:
    gitlab-registry-svc:
      loadBalancer:
        servers:
          - url: 'https://192.168.11.28:5050' 

The middlewares just forwards real IP.

I rewrote the gitlab.rb file carefully from the original auto-generated one to try and remove any erroneous mistakes in the last one, and I’m now getting the following error:

Error response from daemon: Get https://registry.example.com/v2/: unable to decode token response: invalid character '<' looking for beginning of value

But, if I browse directly it does look like there are some end points serving correctly at the address:

https://registry.example.com/v2/

errors	
0	
code	"UNAUTHORIZED"
message	"authentication required"
detail	null

https://registry.example.com/v2/_catalog

errors	
0	
code	"UNAUTHORIZED"
message	"authentication required"
detail	
0	
Type	"registry"
Class	""
Name	"catalog"
Action	"*"

The auth errors don’t look concerning, as, I haven’t authed to view the output, so it makes sense and looks normal?

Success!!

I did some further ‘light’ reading after stumbling across the v2 API working, and found a great blog by Pim Widdershoven about retrieving authentication tokens from the API.

I tried the commands from his blog:

curl --user 'username:password' 'https://gitlab.domain.com/jwt/auth?client_id=docker&offline_token=true&service=container_registry&scope=repository:your-repo-name:push,pull'

This gave me another error, but one I am more familiar with:

Found. Redirecting to https://auth.example.com?rd=https%3A%2F%2Fgitlab.example.com%2Fjwt%2Fauth%3Fclient_id%3Ddocker%26offline_token%3Dtrue%26service%3Dcontainer_registry%26scope%3Drepository%3Atext-compare%3Apush%2Cpull&rm=GET

Which isn’t really an error at all, but my authentication front end Authelia doing its job a little too well. With the front end removed from gitlab.example.com, I was able to login to the registry and push a container just fine:

The push refers to repository [registry.example.com/tom/text-compare]
f1de419122b4: Pushed 
31617a00cd25: Pushed 
6a641a0c23c5: Pushed 
a79fe6dff072: Pushed 
87deea508850: Pushed 
90c4db1d5ef5: Pushed 
cd7100a72410: Pushed 

This leaves me with the choice of not protecting/integrating with Authelia (OpenID) for GitLab, or just not using GitLab credentials to login to the repo (and maybe not using the built in repo, but connecting it to GitLab).

Either way, issue resolved! Thanks very much for your help, it was helpful to push in the right directions, I appreciate it!

image1337×347 31.4 KB

Glad you could solve it on your own, and we learned some new ways of debugging and researching

Absolutely! Thanks for the encouraging replies that go through your thought process, very helpful to develop my own analytical skills better!