Troubles enabling Container Registry behind Traefik reverse proxy

How to Use GitLab
,
1

Hi,

I’ve started putting up my self-hosted Gitlab instance and I’m putting it behind a Traefik2 reverse proxy. Everything is working, I can reach gitlab just fine and the ssh connection also works. Gitlab runners are already active.

However now I’m trying to get the container registry up and running and I have not been able to do that so I’m going to the experts.

System
I use the latest version of Gitlab-ce via docker-compose. Below you will find the contents of my compose file.

Normally most of my stuff is behind an OAUTH instance but in this case I already tried to not put the registry behind that.

  gitlab:
    image: gitlab/gitlab-ce:latest
    container_name: gitlab
    networks:
      home_internal:
        ipv4_address: 192.168.89.252 # You can specify a static IP
    security_opt:
      - no-new-privileges:true
    restart: always
    hostname: gitlab.$DOMAINNAME_CLOUD_SERVER
    volumes:
      - $DOCKERDIR/appdata/gitlab/data:/var/opt/gitlab
      - $DOCKERDIR/appdata/gitlab/logs:/var/log/gitlab
      - $DOCKERDIR/appdata/gitlab/config:/etc/gitlab
    secrets:
      - github_omniauth_app_id
      - github_omniauth_app_secret
    environment:
      GITLAB_OMNIBUS_CONFIG: |
        external_url 'http://gitlab.$DOMAINNAME_CLOUD_SERVER'
        nginx['listen_port'] = 80
        nginx['listen_https'] = false
        nginx['proxy_set_headers'] = {
          "X-Forwarded-Proto" => "https",
          "X-Forwarded-Ssl" => "on"
        }
        registry_external_url 'http://registry.$DOMAINNAME_CLOUD_SERVER'
        registry['enable'] = true
        registry_nginx['listen_https'] = false
        registry_nginx['listen_port'] = 5005
        registry_nginx['proxy_set_headers'] = {
          "X-Forwarded-Proto" => "https",
          "X-Forwarded-Ssl" => "on"
        }
    labels:
      - "traefik.enable=true"
      ## HTTP Routers
      - "traefik.http.routers.gitlab-rtr.entrypoints=https"
      - "traefik.http.routers.gitlab-rtr.rule=Host(`gitlab.$DOMAINNAME_CLOUD_SERVER`)"
      ## Middlewares
      - "traefik.http.routers.gitlab-rtr.middlewares=chain-oauth@file"
      ## HTTP Services
      - "traefik.http.routers.gitlab-rtr.service=gitlab-svc"
      - "traefik.http.services.gitlab-svc.loadbalancer.server.port=80"
      # To ensure ssh works
      - 'traefik.tcp.routers.gitlab-ssh.entrypoints=ssh'
      - 'traefik.tcp.routers.gitlab-ssh.rule=HostSNI(`*`)'
      - 'traefik.tcp.routers.gitlab-ssh.service=gitlab-ssh-svc'
      - 'traefik.tcp.services.gitlab-ssh-svc.loadbalancer.server.port=22'
      ## HTTP Routers
      - "traefik.http.routers.gitlab-registry.entrypoints=https"
      - "traefik.http.routers.gitlab-registry.rule=Host(`registry.$DOMAINNAME_CLOUD_SERVER`)"
      ## Middlewares
      - "traefik.http.routers.gitlab-registry.middlewares=chain-no-auth@file"
      ## HTTP Services
      - "traefik.http.routers.gitlab-registry.service=gitlab-registry-svc"
      - "traefik.http.services.gitlab-registry-svc.loadbalancer.server.port=5005"

What have I tried
I have tried various settings combinations for the GITLAB_OMNIBUS_CONFIG but I cannot find the correct working one.

With the example above when I do a docker login -u USER -p TOKEN registry.DOMAIN.com I end up with a:

Error response from daemon: Get "https://registry.DOMAIN.com/v2/": unable to decode token response: invalid character '<' looking for beginning of value

I already searched the forum for various topics on this subject but they have not given me the right answer yet unfortunately.

Topics like:

Thanks for taking the time to be thorough in your request, it really helps! :blush:

2

Did you ever figure this one out? I am trying to get the registry to work and it is clearly enabled; I can reach it on the gitlab web client, but I cannot docker login and get a time-out error each time I try.

3

No I never did. I just dropped selfhosting gitlab as a solution and moved to a different system (gitea). It was a combination of this being unclear, and that gitlab is quite heavy to run compared to gitea in terms of server resources.

1 Like
4

I have figured it out! I have it working finally and I could not be happier :partying_face:

No need to reconf traefik, just had to get the labels in my gitlab docker compose to be correct.

Now I have this in my GITLAB_OMNIBUS_CONFIG:

        # Container Registry
        #here as a side note; my dns has a *.gitlab wildcard that redirects to the same ip as gitlab.example.com
        registry_external_url 'https://registry.gitlab.example.com' 
        # Enable registry
        registry['enable'] = true
        # Allow registry features in Gitlab UI
        gitlab_rails['registry_enabled'] = true
        # Allow Gitlab's internal NGINX to handle traffic for the registry
        registry_nginx['enable'] = true
        registry_nginx['listen_port'] = 5050
        registry_nginx['listen_https'] = false

and then in my labels I have:

      # Container Registry
      - "traefik.http.routers.registry.rule=Host(`registry.gitlab.example.com`)"
      - "traefik.http.routers.registry.entrypoints=websecure"
      - "traefik.http.routers.registry.tls=true"
      - "traefik.http.routers.registry.tls.certresolver=letsencrypt"
      - "traefik.http.routers.registry.service=registry"
      - "traefik.http.services.registry.loadbalancer.server.port=5050"

I think it was extremely important to have all three registry['enable'] = true, gitlab_rails['registry_enabled'] = true, and registry_nginx['enable'] = true and to undestand what they actually do. Took me a while bit I got there in the end. Hope this might help someone else as I found mupltiple threads about this but no clear answer anywhere as to why people’s configs are not working.

1 Like