Container registry behind reverse proxy

Hi,

I’m hosting a Gitlab CE run out docker for a small development firm. They’re set up with CI right now to push to dockerhub right now. Customer has asked if it is possible to use the registry functionality of GitLab itself.

The setup seems straightforward, however there are a lot of blanks when you’re running behind a reverse proxy.

The domain is using Cloudflare for the SSL and I’ve configured my cloudflare origin certificates in my reverse proxy. The communication between the reverse proxy (rancher haproxy) and the gitlab container is http.
This is done by using the following parameters in omnibus
nginx[‘listen_port’] = 80
nginx[‘listen_https’] = false
external_url ‘https://gitlab.customer.com’

Are there similar parameters to set up the container registry through a reverse proxy that strips the SSL?

Thanks in advance!

I think you can do something similar :

  registry_nginx['listen_port'] = 5000 
  registry_nginx['listen_https'] = false
  registry_nginx['proxy_set_headers'] = {
    "X-Forwarded-Proto" => "https",
    "X-Forwarded-Ssl" => "on"
  }

This is what I used for my instance.

It works, but in my case (I run gitlab using it’s official all-in-one docker image) it was required to use different port for registry_nginx because 5000 is used by docker registry itself, that gitlab runs.

So, I set:

registry_nginx['listen_port'] = 5005

And run gitlab docker image with additional port mapping 5000:5005 (host:container)

The proxy chain is like that:

[public nginx]:443 -> [docker container port mapping]:5000 -> [gitlab nginx]:5005 -> [gitlab docker registry]:5000

You sir (xak2000) are a life saver!
I was struggeling with setting up the registry, but your hint about your proxy chain solved it!

Thank you!

Hello, thanks for the help! I managed to setup the container registry. I have my own GitLab Server running behind HAProxy. However, now I’m experiencing problems with pushing docker images. I can login to docker registry with docker login external_url, but I cannot login with docker login external_url:registry_port (the registry is set up on the same URL and a chosen port). If I login with docker external_url, I get some http json issues: error parsing HTTP 404 response body: invalid character '<' looking for beginning of value:. I think this is due to the fact that by loggin to the external URL I use the gitlab endpoints instead of the docker ones and that causes header problems. Has anyone experienced the same problem?

I’m having the same issue. Were you able to fix this?

Hello @neilborromeo. Yes, I managed to fix it. The issue was with the Haproxy settings. The requests to external_url:registry_port were not redirected correctly.

These are the settings that have to be added to the haproxy.cfg file:

frontend relation-<registry-port>
    bind *:<registry-port> ssl crt <path-to-pem-certificate>
    acl redirect_registry hdr_beg(host) -i <external-url>
    default_backend redirect_registry

backend redirect_registry
    mode http
    cookie SERVERID insert indirect nocache
    server redirect <gitlab-server-ip>:<registry-port> cookie redirect_registry

After apply the changes with:

sudo systemctl restart haproxy

Hope this helps!

hey, thank you.

I’m assuming that you have something like this in your gitlab.rb?

registry_external_url "https://registry.yourdomain.com:5050"

It’s been a week finding a way to solve this. I can login to docker registry just fine using the gitlab.mydomain.com but not on a custom domain.

Problem with that is I can’t push anything to gitlab.mydomain.com and returns something like this:

60e207725d93: Retrying in 1 second 
dc12975afe82: Retrying in 1 second 
6b8255ab91f5: Retrying in 1 second 
dc1af3d61d7c: Retrying in 1 second 
f03595999890: Retrying in 1 second 
21b804938fb0: Waiting 
db88cebe1d84: Waiting 
26b31c6c191c: Waiting 
eb61a33299ca: Waiting 
c3f11d77a5de: Waiting 
received unexpected HTTP status: 200 OK

Hello,

No, I’m keeping the container registry on the same domain as gitlab. In the gitlab.rb tI have the settings as specified at the beginning of this discussion, with url without port and the ngninx proto forward stuff.

Also, you should login with gitlab.mydomain.com:5050 hence adding the port. Otherwise the HTTP calls will be redirected via the haproxy through the normal domain and the push/pull operation will get unexpected HTTP requests for docker (which translates things in json). You should also tag your image including url and port: docker build -t <docker-registry-url:port>/<project-name>/<repository-name>/<some-tag> for pushing/pulling.

Hope this helps

thanks @giusebar this is somehow not working for me with HAProxy.
Originally, I have my setup behind Cloudflare’s ZTNA and I don’t have the liberty to use https://gitlab.mydomain.com:5050.

I lost interest in finding a solution for this and plan to host a private docker registry. Thank you!

Did you ever figure this one out? I am having the same exact issue, where I can login to my-domain.net but not my-domain.net:5050 with docker login.

As a reply to myself, here is how I got the container registry to work behind traefik:

I have been having trouble with Gitlab’s Container Registry behind a traefik reverse proxy.

My login would work normally but uploading images which have large layers would cause issues, they would be cancelled and would retry the upload and it would fail each time.

I have found that traefik was causing this and more precisely traefik’s respondingTimeouts

Because the image had a large layer and it would take more time to upload (a few minutes), the upload would get cancelled after 60 seconds (default traefik config). I have increased this to 10 minutes and everything works well now.

These are my labels

      - "--entryPoints.websecure.address=:443"
      - "--entryPoints.websecure.transport.respondingTimeouts.readTimeout=600"
      - "--entryPoints.websecure.transport.respondingTimeouts.writeTimeout=600"
      - "--entryPoints.websecure.transport.respondingTimeouts.idleTimeout=600"