Problem to solve
I am migrating a self-hosted omnibus install to another server using a dockerized install, with docker compose. I am doing this by restoring a full backup on the new server.
The docker registry works fine on the omnibus install, but not on the new dockerized install.
Apart from that, all of gitlab seems to be working fine.
Symptoms of the problem (probably?):
- I am not seeing any tags in the Registry under any project, always shows
0 tags. - Clicking Registry under any project results in a very long load, >20s
- I used Gitlab’s performance bar to try and understand why the page load is so slow and, strangely, I am seeing a request begin made to http://gitlab/api/graphql. Not seeing this on my omnibus server.
- There is also a request to /api/graphql that take over 10s.
Under Chrome dev tools, Network tab, I am also seeing a CORS error related to the graphql API:
The request with the CORS error tries to make a graphql query using getProjectContainerRepositories, and that fails (which I guess explains partly why the container Registry isn’t working?)
I looked at the storage and all data under container_registry/docker/registry seems to be present on my new server, same as my old omnibus install server.
There are some differences in directory ownerships, I am not sure why:
| Directory | Omnibus install | Dockerized install |
|---|---|---|
| container_registry | drwxrwx--- 3 registry git |
drwxrwx--- 3 registry git |
| container_registry/docker | drwxr-xr-x 3 registry registry |
drwxr-xr-x 3 registry git |
| container_registry/docker/registry | drwxr-xr-x 3 registry registry |
drwxr-xr-x 3 registry git |
| container_registry/docker/registry/v2 | drwxr-xr-x 4 registry registry |
drwxr-xr-x 3 registry git |
| … | … | … |
I let the gitlab-backup restore command handle the permissions and ownership of the registry data.
Steps to reproduce, steps taken to debug
I have followed as best I can:
- Install GitLab by using Docker | GitLab
- Troubleshooting GitLab installations that use Docker | GitLab
- Restore GitLab | GitLab
- GitLab container registry administration | GitLab
I made sure I am able to connect to the docker registry using docker login gitlab.example.com:5050. The command completes successfully and instantly, unlike loading a Registry page in the frontend.
As stated above, I used Gitlab’s performance bar and Chrome dev tools to try and figure out the reason behind the slow registry page load.
Configuration
I am using docker compose with the nginx-proxy image as a reverse proxy to get gitlab-runner, gitlab-pages and the container registry working on the same host.
docker-compose.yml:
services:
gitlab:
image: gitlab/gitlab-ce:16.11.3-ce.0
container_name: gitlab
restart: always
hostname: "gitlab.example.com"
environment:
GITLAB_OMNIBUS_CONFIG: |
# Add any other gitlab.rb configuration here, each on its own line
external_url 'http://gitlab.example.com'
gitlab_rails['gitlab_shell_ssh_port'] = 22
gitlab_rails['backup_path'] = "/mnt/gitlab/gitlab_backups"
gitlab_rails['backup_keep_time'] = 604800 # 7 days
git_data_dirs({
"default" => {
"path" => "/mnt/gitlab/git-data"
}
})
gitlab_rails['env'] = {
'http_proxy' => 'http://10.128.27.6:8080',
'https_proxy' => 'http://10.128.27.6:8080',
'HTTP_PROXY' => 'http://10.128.27.6:8080',
'HTTPS_PROXY' => 'http://10.128.27.6:8080',
'no_proxy' => 'localhost,127.0.0.0/8,::1,.example.com',
'NO_PROXY' => 'localhost,127.0.0.0/8,::1,.example.com'
}
gitlab_rails['trusted_proxies'] = ['192.168.1.1', 'nginx-proxy', 'gitlab.example.com']
registry_external_url 'https://gitlab.example.com:5050'
gitlab_rails['registry_enabled'] = true
gitlab_rails['registry_path'] = "/mnt/gitlab/container_registry"
registry['enable'] = true
registry['registry_http_addr'] = "192.168.1.2:5000" # very important, does not work without
registry_nginx['enable'] = false
registry['env'] = {
"REGISTRY_HTTP_RELATIVEURLS" => true
}
prometheus['enable'] = false
expose:
- 80 # http
- 443 # https
- 22 # gitlab SSH
- 8090 # gitlab-pages
- 5000 # gitlab docker registry
volumes:
- $GITLAB_HOME/config:/etc/gitlab:rw
- $GITLAB_HOME/logs:/var/log/gitlab:rw
- $GITLAB_HOME/data:/var/opt/gitlab:rw
- /mnt/gitlab:/mnt/gitlab:rw
shm_size: 256m
networks:
proxynet:
ipv4_address: 192.168.1.2
gitlab-runner:
image: gitlab/gitlab-runner:alpine
container_name: gitlab-runner
restart: always
volumes:
- ./certs:/etc/gitlab-runner/certs:ro
- /srv/gitlab-runner/config:/etc/gitlab-runner
- /var/run/docker.sock:/var/run/docker.sock
networks:
proxynet:
ipv4_address: 192.168.1.3
nginx-proxy:
# https://github.com/nginx-proxy/nginx-proxy
# https://github.com/nginx-proxy/nginx-proxy/tree/main/docs#ssl-support
image: nginxproxy/nginx-proxy:1.6
container_name: nginx-proxy
hostname: nginx-proxy
restart: always
environment:
TZ: America/Toronto
SSL_POLICY: Mozilla-Modern
ports:
- "80:80" # http
- "443:443" # https
- 22:22 # gitlab ssh
- 5050:5050 # gitlab docker registry
volumes:
- /var/run/docker.sock:/tmp/docker.sock:ro
- ./certs:/etc/nginx/certs:ro
- ./nginx/proxy.conf:/etc/nginx/conf.d/proxy.conf:ro
networks:
proxynet:
ipv4_address: 192.168.1.1
networks:
proxynet:
ipam:
driver: default
config:
- subnet: "192.168.0.0/16"
And the nginx.conf used for nginx-proxy:
upstream gitlab {
server 192.168.1.2:80;
}
upstream docker-registry {
server 192.168.1.2:5000;
}
upstream gitlab-pages {
server 192.168.1.2:8090;
}
server {
server_name example.gitlab.com;
access_log /var/log/nginx/access.log vhost;
http2 on;
listen 80 ;
location /.well-known/acme-challenge/ {
auth_basic off;
allow all;
root /usr/share/nginx/html;
try_files $uri =404;
break;
}
listen 443 ssl ;
ssl_protocols TLSv1.2 TLSv1.3;
ssl_ciphers ECDHE-ECDSA-AES256-GCM-SHA384;
ssl_conf_command Ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256;
ssl_prefer_server_ciphers off;
ssl_session_timeout 5m;
ssl_session_cache shared:SSL:50m;
ssl_session_tickets off;
ssl_certificate /etc/nginx/certs/example.gitlab.com.crt;
ssl_certificate_key /etc/nginx/certs/example.gitlab.com.key;
location / {
proxy_pass http://gitlab;
proxy_set_header X-Forwarded-For $remote_addr;
set $upstream_keepalive false;
}
}
# Inspired from /var/opt/gitlab/nginx/conf/gitlab-registry.conf on a gitlab container
# and https://stackoverflow.com/questions/47584834/gitlab-docker-registry-with-external-nginx-and-omnibus
#
# This proxies directly to the docker registry running on the gitlab container.
# The gitlab container must be configured as such:
# registry_external_url 'https://host.domain:5050'
# gitlab_rails['registry_enabled'] = true
# registry['enable'] = true
# registry['registry_http_addr'] = "<gitlab container IP>:5000" # very important, does not work without
# registry_nginx['enable'] = false
server { ## HTTPS server
listen *:5050 ssl;
server_name example.gitlab.com;
server_tokens off; ## Don't show the nginx version number, a security best practice
client_max_body_size 0;
chunked_transfer_encoding on;
## Strong SSL Security
## https://raymii.org/s/tutorials/Strong_SSL_Security_On_nginx.html & https://cipherli.st/
ssl_certificate /etc/nginx/certs/example.gitlab.com.crt;
ssl_certificate_key /etc/nginx/certs/example.gitlab.com.key;
ssl_ciphers 'ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384';
ssl_protocols TLSv1.2 TLSv1.3;
ssl_prefer_server_ciphers off;
ssl_session_cache shared:SSL:50m;
ssl_session_tickets off;
ssl_session_timeout 1d;
## Real IP Module Config
## http://nginx.org/en/docs/http/ngx_http_realip_module.html
## HSTS Config
## https://www.nginx.com/blog/http-strict-transport-security-hsts-and-nginx/
add_header Strict-Transport-Security "max-age=63072000";
access_log /var/log/nginx/gitlab_registry_access.log;
error_log /var/log/nginx/gitlab_registry_error.log;
location / {
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto https;
proxy_set_header X-Forwarded-Ssl on;
proxy_read_timeout 900;
proxy_cache off;
proxy_buffering off;
proxy_request_buffering off;
proxy_http_version 1.1;
proxy_pass http://docker-registry;
}
}
# Inspired from /var/opt/gitlab/nginx/conf/gitlab-pages.conf on a gitlab container
server {
listen *:80;
server_name ~^(?<group>.*)\.docs\.example\.com$;
server_tokens off; ## Don't show the nginx version number, a security best practice
## Disable symlink traversal
disable_symlinks on;
## Real IP Module Config
## http://nginx.org/en/docs/http/ngx_http_realip_module.html
## HSTS Config
## https://www.nginx.com/blog/http-strict-transport-security-hsts-and-nginx/
add_header Strict-Transport-Security "max-age=63072000";
## Individual nginx logs for this GitLab vhost
access_log /var/log/nginx/gitlab_pages_access.log;
error_log /var/log/nginx/gitlab_pages_error.log;
# Define custom error pages
error_page 403 /403.html;
error_page 404 /404.html;
# Pass everything to pages daemon when namespace in host
location / {
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto http;
proxy_hide_header X-Gitlab-Namespace-In-Path;
# Prevent NGINX from caching pages in response to the pages `Cache-Control`
# header.
#
# Browsers already respect this directive and Pages can handle the request
# volume without help from NGINX.
#
# If this changes in the future, ensure `proxy_cache_key` is set to a value
# like `$scheme$host$request_uri`, as the default value does not take the
# Pages hostname into account, leading to incorrect responses being served.
#
# See https://gitlab.com/gitlab-org/gitlab-pages/issues/73
proxy_cache off;
proxy_http_version 1.1;
proxy_pass http://gitlab-pages;
}
}
Versions
- Self-managed
- GitLab.com SaaS
GitLab v16.11.3-ce
(I know this isn’t the latest and has security issues – it is why I am migrating to a new server, to be able to upgrade. I cannot upgrade my omnibus install further as it is installed on Ubuntu 18.04, which is not supported anymore.)