Container Registry doesn't work with dockerized gitlab behind https

Hello,

I’m running a self-hosted dockerized Gitlab and gitlab runner on a VPS having a NGINX.
I have no problem to run Gitlab and set the certificate so that it’s accessible through https.
But for some reason I can’t make the registry work.

Here is the docker-compose.yml file:

version: '3'

services:
  gitlab:
    image: gitlab/gitlab-ce:latest
    container_name: gitlab
    hostname: EXAMPLE.COM
    ports:
      - "4433:443"
      - "8080:80"
      - "2222:22"
      - "5005:5005"
    volumes:
      - /data/gitlab/config:/etc/gitlab
      - /data/gitlab/logs:/var/log/gitlab
      - /data/gitlab/data:/var/opt/gitlab
      - /etc/letsencrypt/live/registry.EXAMPLE.COM:/etc/letsencrypt/live/registry.EXAMPLE.COM
    restart: always
    networks:
      - gitlab
  gitlab-runner:
    image: gitlab/gitlab-runner:alpine
    container_name: gitlab-runner
    restart: always
    depends_on:
      - gitlab
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - /data/gitlab-runner:/etc/gitlab-runner
    networks:
      - gitlab

networks:
  gitlab:
    name: gitlab-network

And here is my NGINX configuration in “sites-enabled”:

upstream docker-registry {
  server 127.0.0.1:8080;
}

server {
  server_name EXAMPLE.COM;
  client_max_body_size 50m;
  location / {
    proxy_pass  http://127.0.0.1:8080/;
    proxy_set_header Host $http_host;
    proxy_http_version 1.1;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_set_header X-Real-IP $remote_addr;
  }
    listen 443 ssl; # managed by Certbot
    ssl_certificate /etc/letsencrypt/live/EXAMPLE.COM/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/EXAMPLE.COM/privkey.pem; # managed by Certbot
    include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot

}
server {
    if ($host = EXAMPLE.COM) {
        return 301 https://$host$request_uri;
    } # managed by Certbot


  listen 80;
  server_name EXAMPLE.COM;
    return 404; # managed by Certbot

}


server {
  server_name registry.EXAMPLE.COM;
  client_max_body_size 50m;
  
  location / {
    proxy_pass  http://127.0.0.1:5005/;
    proxy_set_header Host $http_host;
    proxy_http_version 1.1;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_set_header X-Real-IP $remote_addr;
  }
    listen 443 ssl; # managed by Certbot
    ssl_certificate /etc/letsencrypt/live/registry.EXAMPLE.COM/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/registry.EXAMPLE.COM/privkey.pem; # managed by Certbot
}

If I only set

registry_external_url 'https://registry.EXAMPLE.COM'
gitlab_rails['registry_enabled'] = true
gitlab_rails['registry_host'] = "registry.EXAMPLE.COM"

The container doesn’t run properly and I have this error:

==> /var/log/gitlab/nginx/current <==
2023-12-06_06:47:27.94903 nginx: [emerg] cannot load certificate "/etc/gitlab/ssl/registry.EXAMPLE.COM.crt": BIO_new_file() failed (SSL: error:02001002:system library:fopen:No such file or directory:fopen('/etc/gitlab/ssl/registry.EXAMPLE.COM.crt','r') error:2006D080:BIO routines:BIO_new_file:no such file)

I than add these lines in the gitlab.rb file:

registry_nginx['ssl_certificate'] = "/etc/letsencrypt/live/registry.EXAMPLE.COM/fullchain.pem
registry_nginx['ssl_certificate_key'] = "/etc/letsencrypt/live/registry.EXAMPLE.COM/privkey.pem
nginx['ssl_certificate'] = "/etc/letsencrypt/live/registry.EXAMPLE.COM/fullchain.pem
nginx['ssl_certificate_key'] = "/etc/letsencrypt/live/registry.EXAMPLE.COM/privkey.pem

But for some reason I still have an error:

==> /var/log/gitlab/nginx/current <==
2023-12-06_06:56:03.47156 nginx: [emerg] cannot load certificate "/etc/letsencrypt/live/registry.EXAMPLE.COM/fullchain.pem": BIO_new_file() failed (SSL: error:02001002:system library:fopen:No such file or directory:fopen('/etc/letsencrypt/live/registry.EXAMPLE.COM/fullchain.pem','r') error:2006D080:BIO routines:BIO_new_file:no such file)

Although when I check inside the container, using docker exec -it bash, the ls -las command shows that the files are found.

I spent so much time on it, I think I read all the questions and articles related to my problem, but I can’t find a solution yet. I think I’m close but I don’t understand what goes wrong.
If anybody could give me a hand, that would be great, thanks in advance.

Can you share the full output of the ls command? Maybe it is a file permission problem, leading to the not found error above.

Sorry for the late of my reply, I didn’t receive any notification.
Here is the result of the command:

xxx@HOSTMACHINE:~$ docker exec -it CONTAINER_HASH bash
root@gitlab:/# ls -las /etc/letsencrypt/live/registry.EXAMPLE.COM
total 12
4 drwxr-xr-x 2 root root 4096 Dec  5 23:47 .
4 drwxr-xr-x 3 root root 4096 Dec  6 07:08 ..
4 -rw-r--r-- 1 root root  692 Dec  5 23:47 README
0 lrwxrwxrwx 1 root root   50 Dec  5 23:47 cert.pem -> ../../archive/registry.EXAMPLE.COM
/cert1.pem
0 lrwxrwxrwx 1 root root   51 Dec  5 23:47 chain.pem -> ../../archive/registry.EXAMPLE.COM
/chain1.pem
0 lrwxrwxrwx 1 root root   55 Dec  5 23:47 fullchain.pem -> ../../archive/registry.EXAMPLE.COM
fullchain1.pem
0 lrwxrwxrwx 1 root root   53 Dec  5 23:47 privkey.pem -> ../../archive/registry.EXAMPLE.COM
/privkey1.pem

Note: the 4 lines starting with “0” have the links (the part “cert.pem → …/…/archive/registry.EXAMPLE.COM
/cert1.pem” for example) in red. I don’t know if it has a particular meaning.

I know the 777 rights is not a good practice, but I woulad have thought granting all the permissions would lead me somewhere.

Thanks in advance if you have any advice.