gitlab registry: http: server gave HTTP response to HTTPS client

I am fiddling around since days to get gitlab’s registry running and accessible without any luck.
I tried different variants based on gitlab docs, issues found here and other resources on the internet, but
I think I may have run into either missleading information or potentially hit some bugs.

Setup/Situation:

  • on-premise / internal installation
  • Docker 24.0.1 running on Ubuntu 22.04.2 LTS (x86_64)
  • GitLab Community Edition v16.0.1 installed via docker compose (see docker-compose.yml below)
  • SSL for gitlab webserver configured properly and working (using company’s wildcard certificate)
  • gitlab-runner as docker container setup and working

Issue Description

  • in the end I want to have my .gitlab-ci.yml setup to create image and deploy to registry
  • I approached this by manually running “docker login gitlab.mycompany.com:5050” as shown in “Container Registry” page within my gitlab project
  • but this results in:

    Error response from daemon: Get “https://gitlab.mycompany.com:5050/v2/”: http: server gave HTTP response to HTTPS client

  • so I edited /etc/docker/daemon.json (as seen in some examples)

    { “insecure-registries”:[“gitlab.mycompany.com:5050”] }

  • which leads to this when running same docker login command:

    Error response from daemon: Get “http://gitlab.mycompany.com:5050/v2/”: Get “/jwt/auth?account=aa&client_id=docker&offline_token=true&service=container_registry”: unsupported protocol scheme “”

My understandings so far (might be wrong, but that’s what I tried myself and figured out)

  • gitlab provides its own repository server
  • this repository server runs per default on 172.0.0.1:5000 and is not ssl encrypted
  • due to I only have 1 IP address and 1 DNS name from “outside” to my gitlab containers I have to use nginx as reverse and ssl proxy for gitlab’s registry service
  • First I tried to have all gitlab settings (gitlab.rb) set via my docker-compose.yml, then I switched to have all settings done inside /etc/gitlab/gitlab.rb and cleared my docker-compose.yml
  • When I used registry[‘registry_http_addr’] = “127.0.0.1:5000” I did not see any service on port 5000 (netstat -tulpn did not show a line for port 5000.
  • When I used registry[‘registry_http_addr’] = “0.0.0.0:5000” I do see listening port 5000 but only on tcp6

What makes me wonder

  • assuming nginx configuration files got generated correctly I do not see any ssl specific settings within /var/opt/gitlab/nginx/conf/gitlab-registry.conf
  • I would assume at least some sort of “listen *:5050 ssl http2;” as well as “ssl_certificate” and “ssl_certificate_key” but I cannot find these settings inside gitlab-registry.conf

My config

  • docker-compose.yml:
services:
  web:
    image: 'gitlab/gitlab-ce:latest'
    container_name: gitlab
    restart: always
    hostname: 'gitlab.mycompany.com'
    ports:
      - '10.50.7.215:80:80'
      - '10.50.7.215:443:443'
      - '10.50.7.215:2222:22'
      - '10.50.7.215:5050:5050'
      - '10.50.7.215:5000:5000' # for testing only
    volumes:
      - '$GITLAB_HOME/config:/etc/gitlab'
      - '$GITLAB_HOME/logs:/var/log/gitlab'
      - '$GITLAB_HOME/data:/var/opt/gitlab'
    shm_size: '256m'
  • /etc/gitlab/gitlab.rb:
### GENERAL SETTINGS
###
external_url 'https://gitlab.mycompany.com'
nginx['ssl_certificate'] = "/etc/gitlab/ssl/WA/wildcard_mycompany.crt_ca-bundle"
nginx['ssl_certificate_key'] = "/etc/gitlab/ssl/WA/wildcard_mycompany-automotive.key"


###
### REGISTRY SETTINGS
###
registry['enable'] = true
registry['registry_http_addr'] = "0.0.0.0:5000"
registry_external_url = "https://gitlab.mycompany.com:5050"
registry_nginx['redirect_http_to_https'] = true
registry_nginx['enable'] = true
registry_nginx['listen_port'] = 5050
registry_nginx['ssl_certificate'] =  "/etc/gitlab/ssl/WA/wildcard_mycompany.crt_ca-bundle"
registry_nginx['ssl_certificate_key'] = "/etc/gitlab/ssl/WA/wildcard_mycompany.key"
gitlab_rails['registry_enabled'] = true
gitlab_rails['registry_host'] = "gitlab.mycompany.com"
gitlab_rails['registry_port'] = "5050"
gitlab_rails['registry_path'] = "/var/opt/gitlab/gitlab-rails/shared/registry"
  • /var/opt/gitlab/nginx/conf/gitlab-registry.conf:
  listen *:5050;
  server_name gitlab.mycompany.com;
  server_tokens off; ## Don't show the nginx version number, a security best practice

  client_max_body_size 0;
  chunked_transfer_encoding on;


  ## Real IP Module Config
  ## http://nginx.org/en/docs/http/ngx_http_realip_module.html

  ## HSTS Config
  ## https://www.nginx.com/blog/http-strict-transport-security-hsts-and-nginx/
  add_header Strict-Transport-Security "max-age=63072000";

  access_log  /var/log/gitlab/nginx/gitlab_registry_access.log gitlab_access;
  error_log   /var/log/gitlab/nginx/gitlab_registry_error.log error;


  location / {

    proxy_set_header Host $http_host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;

    proxy_read_timeout                  900;
    proxy_cache off;
    proxy_buffering off;
    proxy_request_buffering off;
    proxy_http_version 1.1;

    proxy_pass          http://0.0.0.0:5000;
  }


} ## end HTTPS server

I did try around so many different settings and configurations but in the end login in to registry isn’t working. So maybe some of you may either guide me into solution or correct my understandings and expectations.
Either way, your support on this really highly appreciated

regards
Joerg

If your GitLab server is not behind another LoadBalancer or revers proxy only things you need to add to gitlab.rb are

registry['enable'] = true
registry_external_url 'https://gitlab.mycompany.com:5050'
registry_nginx['ssl_certificate'] =  "/etc/gitlab/ssl/WA/wildcard_mycompany.crt_ca-bundle"
registry_nginx['ssl_certificate_key'] = "/etc/gitlab/ssl/WA/wildcard_mycompany.key"

anything else should be removed.

You made my day, I have found my configuration error:

I used:
registry_external_url =https://gitlab.mycompany.com:5050’ instead of
registry_external_url ‘https://gitlab.mycompany.com:5050

so this complete mess was due to an extra equal sign which seemed to me to look fine

Regards
Joerg

1 Like