Registry shown as available in projects but host is not listening on 5005 and project cant connect to it

Belutak · March 8, 2023, 4:23pm

Am trying to enable registry on my gitlab-ce instance but connection to it is not working, neither there are errors.
My setup is linux server, with gitlab-ce docker container and registry enabled in it(no separate container for registry).

In my gitlab instance, after enabling registry, i see it as enabled. When i go to my project and than - packages and registries/registry it shows me the ways to connect to in, in my case it is:

docker login registry.gitlab2.arma:5005
docker build -t registry.gitlab2.arma:5005/aspen-developers/aspen-server .
docker push registry.gitlab2.arma:5005/aspen-developers/aspen-server

What i dont see, and what i think is problem, is my host listening on that port. I see on linux ports 80, 443 and 22(those ports are coming from gitlab container) but i do not see 5005 or 5050, and i cant telnet to ip-host:port.
But also when i try to connect to repository with docker login registry.gitlab2.arma:5005 i get connection refused. Small not about gitlab, if you try completely imaginary url, e.g. docker login 550.550.550.550 it will not reject but it will ask you for user pass, i find that making debugging even more challenging.

gitlab version is 15.9

what i find problematic is that there is nothing in logs about this refused connection(which i understand since my host is not listening on port 5005 and connection never comes to it), but there is nothing in logs about nginx not starting the registry or reason why it does not listen on that port or anything like that, so i cannot find anywhere what the exact problem is.
I entered container to check it and its not listening on loopback:5005(so i dont see it on host), it just not listening on that port.
I have found nginx conf file for registry and it seams ok.

registry related part of gitlab.rb

registry_external_url 'https://registry.gitlab2.arma:5005'

gitlab_rails['registry_enabled'] = true
gitlab_rails['registry_host'] = "registry.gitlab2.arma"
gitlab_rails['registry_port'] = "5005"
gitlab_rails['registry_path'] = "/var/opt/gitlab/gitlab-rails/shared/registry"
gitlab_rails['registry_api_url'] = "http://localhost:5000"
gitlab_rails['registry_key_path'] = "/etc/gitlab/ssl/ssl_registry.key"

registry['enable'] = true
registry['token_realm'] = "https://gitlab.mydomain.com:443"
registry['registry_http_addr'] = "localhost:5000"
registry['log_directory'] = "/var/log/gitlab/registry"
registry['env_directory'] = "/opt/gitlab/etc/registry/env"
registry['env'] = {
registry['rootcertbundle'] = "/etc/gitlab/ssl/ssl_registry.crt"
registry_nginx['enable'] = true
registry_nginx['listen_port'] = 5050
registry_nginx['ssl_certificate'] = "/etc/gitlab/ssl/ca.crt"
registry_nginx['ssl_certificate_key'] = "/etc/gitlab/ssl/ca.key"

please note that ssl keys exist and are on that location.

docker compose:

web:
  image: 'gitlab/gitlab-ce:latest'
  restart: always
  hostname: 'gitlab2.arma'

  environment:
    GITLAB_OMNIBUS_CONFIG: |
      # Add any other gitlab.rb configuration here, each on its own line
      external_url 'https://gitlab2.arma'
      gitlab_rails['gitlab_shell_ssh_port'] = 22
      nginx['redirect_http_to_https'] = false
      nginx['ssl_certificate'] = "/etc/gitlab/ssl/ca.crt"
      nginx['ssl_certificate_key'] = "/etc/gitlab/ssl/ca.key"
      nginx['ssl_dhparam'] = "/etc/gitlab/ssl/dhparams.pem"  

  ports:
    - '443:443'
    - '80:80'
    - '2243:22'
    - '5005:5005'
  volumes:
    - '${GITLAB_HOME}/config:/etc/gitlab'
    - '${GITLAB_HOME}/logs:/var/log/gitlab'
    - '${GITLAB_HOME}/data:/var/opt/gitlab'
    - '${GITLAB_HOME}/config/ssl:/etc/gitlab/ssl'

part of log(probably nginx, i paste it yesterday so am not sure)
edit: shrinked since gitlab wont accept length of this post

SC[36mweb_1  |ESC[0m [2023-03-05T03:47:40+00:00] INFO: Loading cookbooks [gitlab@0.0.1, package@0.1.0, logrotate@0.1.0, postgresql@0.1.0, redis@0.1.0, monitoring@0.1.0, registry@0.1.0, mattermost@0.1.0, consul
@0.1.0, gitaly@0.1.0, praefect@0.1.0, gitlab-kas@0.1.0, gitlab-pages@0.1.0, letsencrypt@0.1.0, nginx@0.1.0, runit@5.1.7, acme@4.1.5, crond@0.1.0]
ESC[36mweb_1  |ESC[0m   - registry (0.1.0)
ESC[36mweb_1  |ESC[0m   * file[/etc/gitlab/ssl/ssl_registry.key] action create[2023-03-05T03:47:47+00:00] WARN: only_if block for file[/etc/gitlab/ssl/ssl_registry.key] returned a string, did you mean to run a command?
ESC[36mweb_1  |ESC[0m [2023-03-05T03:47:47+00:00] INFO: file[/etc/gitlab/ssl/ssl_registry.key] backed up to /opt/gitlab/embedded/cookbooks/cache/backup/etc/gitlab/ssl/ssl_registry.key.chef-20230305034747.339892
ESC[36mweb_1  |ESC[0m [2023-03-05T03:47:47+00:00] INFO: file[/etc/gitlab/ssl/ssl_registry.key] updated file contents /etc/gitlab/ssl/ssl_registry.key
ESC[36mweb_1  |ESC[0m     - update content in file /etc/gitlab/ssl/ssl_registry.key from 086bb7 to 37e8bc
ESC[36mweb_1  |ESC[0m     - suppressed sensitive resource[2023-03-05T03:47:47+00:00] INFO: file[/etc/gitlab/ssl/ssl_registry.key] owner changed to 998
ESC[36mweb_1  |ESC[0m [2023-03-05T03:47:47+00:00] INFO: file[/etc/gitlab/ssl/ssl_registry.key] group changed to 998
ESC[36mweb_1  |ESC[0m     -registry_dir=''
ESC[36mweb_1  |ESC[0m     +registry_dir='/var/opt/gitlab/gitlab-rails/shared/registry'
ESC[36mweb_1  |ESC[0m      registry_user='registry'
ESC[36mweb_1  |ESC[0m      registry_group='registry'
ESC[36mweb_1  |ESC[0m   * template[/var/opt/gitlab/nginx/conf/gitlab-registry.conf] action create[2023-03-05T03:47:54+00:00] INFO: template[/var/opt/gitlab/nginx/conf/gitlab-registry.conf] created file /var/opt/gitlab/nginx/conf/gitlab-registry.conf
ESC[36mweb_1  |ESC[0m     - create new file /var/opt/gitlab/nginx/conf/gitlab-registry.conf[2023-03-05T03:47:54+00:00] INFO: template[/var/opt/gitlab/nginx/conf/gitlab-registry.conf] updated file contents /var/opt/gitlab/nginx/conf/gitlab-registry.conf
ESC[36mweb_1  |ESC[0m     - update content in file /var/opt/gitlab/nginx/conf/gitlab-registry.conf from none to 54b7c9
ESC[36mweb_1  |ESC[0m     --- /var/opt/gitlab/nginx/conf/gitlab-registry.conf   2023-03-05 03:47:54.201715865 +0000
ESC[36mweb_1  |ESC[0m     +++ /var/opt/gitlab/nginx/conf/.chef-gitlab-registry20230305-30-vqsd8k.conf   2023-03-05 03:47:54.201715865 +0000
ESC[36mweb_1  |ESC[0m     +  server_name registry.gitlab2.arma;
ESC[36mweb_1  |ESC[0m     +  access_log  /var/log/gitlab/nginx/gitlab_registry_access.log gitlab_access;
ESC[36mweb_1  |ESC[0m     +  error_log   /var/log/gitlab/nginx/gitlab_registry_error.log error;
ESC[36mweb_1  |ESC[0m     +} ## end HTTPS server[2023-03-05T03:47:54+00:00] INFO: template[/var/opt/gitlab/nginx/conf/gitlab-registry.conf] owner changed to 0
ESC[36mweb_1  |ESC[0m [2023-03-05T03:47:54+00:00] INFO: template[/var/opt/gitlab/nginx/conf/gitlab-registry.conf] group changed to 0
ESC[36mweb_1  |ESC[0m [2023-03-05T03:47:54+00:00] INFO: template[/var/opt/gitlab/nginx/conf/gitlab-registry.conf] mode changed to 644
ESC[36mweb_1  |ESC[0m     +  include /var/opt/gitlab/nginx/conf/gitlab-registry.conf;
ESC[36mweb_1  |ESC[0m Recipe: registry::enable
ESC[36mweb_1  |ESC[0m   * directory[create /var/opt/gitlab/registry] action create[2023-03-05T03:47:54+00:00] INFO: directory[create /var/opt/gitlab/registry] created directory /var/opt/gitlab/registry
ESC[36mweb_1  |ESC[0m     - create new directory /var/opt/gitlab/registry
ESC[36mweb_1  |ESC[0m   * account[Docker registry user and group] action create (up to date)
ESC[36mweb_1  |ESC[0m   * directory[create /var/opt/gitlab/registry and set the owner] action create[2023-03-05T03:47:54+00:00] INFO: directory[create /var/opt/gitlab/registry and set the owner] owner changed to 993
ESC[36mweb_1  |ESC[0m [2023-03-05T03:47:54+00:00] INFO: directory[create /var/opt/gitlab/registry and set the owner] mode changed to 700
ESC[36mweb_1  |ESC[0m     - change owner from 'root' to 'registry'
ESC[36mweb_1  |ESC[0m   * directory[create /var/log/gitlab/registry and set the owner] action create[2023-03-05T03:47:54+00:00] INFO: directory[create /var/log/gitlab/registry and set the owner] created directory /var/log/gitlab/registry
ESC[36mweb_1  |ESC[0m     - create new directory /var/log/gitlab/registry[2023-03-05T03:47:54+00:00] INFO: directory[create /var/log/gitlab/registry and set the owner] owner changed to 993
ESC[36mweb_1  |ESC[0m [2023-03-05T03:47:54+00:00] INFO: directory[create /var/log/gitlab/registry and set the owner] mode changed to 700
ESC[36mweb_1  |ESC[0m     - change owner from '' to 'registry'
ESC[36mweb_1  |ESC[0m   * env_dir[/opt/gitlab/etc/registry/env] action create
ESC[36mweb_1  |ESC[0m     * directory[/opt/gitlab/etc/registry/env] action create[2023-03-05T03:47:54+00:00] INFO: directory[/opt/gitlab/etc/registry/env] created directory /opt/gitlab/etc/registry/env
ESC[36mweb_1  |ESC[0m       - create new directory /opt/gitlab/etc/registry/env
ESC[36mweb_1  |ESC[0m     * file[/opt/gitlab/etc/registry/env/SSL_CERT_DIR] action create[2023-03-05T03:47:54+00:00] INFO: file[/opt/gitlab/etc/registry/env/SSL_CERT_DIR] created file /opt/gitlab/etc/registry/env/SSL_CERT_DIR
ESC[36mweb_1  |ESC[0m       - create new file /opt/gitlab/etc/registry/env/SSL_CERT_DIR[2023-03-05T03:47:54+00:00] INFO: file[/opt/gitlab/etc/registry/env/SSL_CERT_DIR] updated file contents /opt/gitlab/etc/registry/env/SSL_CERT_DIR
ESC[36mweb_1  |ESC[0m       - update content in file /opt/gitlab/etc/registry/env/SSL_CERT_DIR from none to 4f45cf
ESC[36mweb_1  |ESC[0m   * directory[/var/opt/gitlab/gitlab-rails/shared/registry] action create[2023-03-05T03:47:54+00:00] INFO: directory[/var/opt/gitlab/gitlab-rails/shared/registry] created directory /var/opt/gitlab/gitlab-rails/shared/registry
ESC[36mweb_1  |ESC[0m     - create new directory /var/opt/gitlab/gitlab-rails/shared/registry[2023-03-05T03:47:54+00:00] INFO: directory[/var/opt/gitlab/gitlab-rails/shared/registry] owner changed to 993
ESC[36mweb_1  |ESC[0m [2023-03-05T03:47:54+00:00] INFO: directory[/var/opt/gitlab/gitlab-rails/shared/registry] group changed to 998
ESC[36mweb_1  |ESC[0m [2023-03-05T03:47:54+00:00] INFO: directory[/var/opt/gitlab/gitlab-rails/shared/registry] mode changed to 770
ESC[36mweb_1  |ESC[0m     - change owner from '' to 'registry'
ESC[36mweb_1  |ESC[0m   * file[/var/opt/gitlab/registry/gitlab-registry.crt] action create[2023-03-05T03:47:54+00:00] INFO: file[/var/opt/gitlab/registry/gitlab-registry.crt] created file /var/opt/gitlab/registry/gitlab-registry.crt
ESC[36mweb_1  |ESC[0m     - create new file /var/opt/gitlab/registry/gitlab-registry.crt[2023-03-05T03:47:54+00:00] INFO: file[/var/opt/gitlab/registry/gitlab-registry.crt] updated file contents /var/opt/gitlab/registry/gitlab-registry.crt
ESC[36mweb_1  |ESC[0m     - update content in file /var/opt/gitlab/registry/gitlab-registry.crt from none to 7df50c
ESC[36mweb_1  |ESC[0m     - suppressed sensitive resource[2023-03-05T03:47:54+00:00] INFO: file[/var/opt/gitlab/registry/gitlab-registry.crt] owner changed to 993
ESC[36mweb_1  |ESC[0m [2023-03-05T03:47:54+00:00] INFO: file[/var/opt/gitlab/registry/gitlab-registry.crt] group changed to 993
ESC[36mweb_1  |ESC[0m     - change owner from '' to 'registry'
ESC[36mweb_1  |ESC[0m     - change group from '' to 'registry'
ESC[36mweb_1  |ESC[0m   * template[/var/opt/gitlab/registry/config.yml] action create[2023-03-05T03:47:54+00:00] INFO: template[/var/opt/gitlab/registry/config.yml] created file /var/opt/gitlab/registry/config.yml
ESC[36mweb_1  |ESC[0m     - create new file /var/opt/gitlab/registry/config.yml[2023-03-05T03:47:54+00:00] INFO: template[/var/opt/gitlab/registry/config.yml] updated file contents /var/opt/gitlab/registry/config.yml
ESC[36mweb_1  |ESC[0m     - update content in file /var/opt/gitlab/registry/config.yml from none to 562c4d
ESC[36mweb_1  |ESC[0m     --- /var/opt/gitlab/registry/config.yml       2023-03-05 03:47:54.561720134 +0000
ESC[36mweb_1  |ESC[0m     +++ /var/opt/gitlab/registry/.chef-config20230305-30-ydvqlu.yml       2023-03-05 03:47:54.561720134 +0000
ESC[36mweb_1  |ESC[0m     +    service: registry
ESC[36mweb_1  |ESC[0m     +storage: {"filesystem":{"rootdirectory":"/var/opt/gitlab/gitlab-rails/shared/registry"},"cache":{"blobdescriptor":"inmemory"},"delete":{"enabled":true}}
ESC[36mweb_1  |ESC[0m     +    service: container_registry
ESC[36mweb_1  |ESC[0m     +    rootcertbundle: /etc/gitlab/ssl/ssl_registry.crt
ESC[36mweb_1  |ESC[0m     +  disabled: true[2023-03-05T03:47:54+00:00] INFO: template[/var/opt/gitlab/registry/config.yml] owner changed to 993
ESC[36mweb_1  |ESC[0m [2023-03-05T03:47:54+00:00] INFO: template[/var/opt/gitlab/registry/config.yml] mode changed to 644
ESC[36mweb_1  |ESC[0m     - change owner from '' to 'registry'
ESC[36mweb_1  |ESC[0m [2023-03-05T03:47:54+00:00] INFO: template[/var/opt/gitlab/registry/config.yml] not queuing delayed action restart on runit_service[registry] (delayed), as it's already been queued
ESC[36mweb_1  |ESC[0m   * service[registry] action nothing (skipped due to action :nothing)
ESC[36mweb_1  |ESC[0m   * runit_service[registry] action enable
ESC[36mweb_1  |ESC[0m     * directory[/opt/gitlab/sv/registry] action create[2023-03-05T03:47:54+00:00] INFO: directory[/opt/gitlab/sv/registry] created directory /opt/gitlab/sv/registry
ESC[36mweb_1  |ESC[0m       - create new directory /opt/gitlab/sv/registry[2023-03-05T03:47:54+00:00] INFO: directory[/opt/gitlab/sv/registry] owner changed to 0
ESC[36mweb_1  |ESC[0m [2023-03-05T03:47:54+00:00] INFO: directory[/opt/gitlab/sv/registry] group changed to 0
ESC[36mweb_1  |ESC[0m [2023-03-05T03:47:54+00:00] INFO: directory[/opt/gitlab/sv/registry] mode changed to 755
ESC[36mweb_1  |ESC[0m     * template[/opt/gitlab/sv/registry/run] action create[2023-03-05T03:47:54+00:00] INFO: template[/opt/gitlab/sv/registry/run] created file /opt/gitlab/sv/registry/run
ESC[36mweb_1  |ESC[0m       - create new file /opt/gitlab/sv/registry/run[2023-03-05T03:47:54+00:00] INFO: template[/opt/gitlab/sv/registry/run] updated file contents /opt/gitlab/sv/registry/run
ESC[36mweb_1  |ESC[0m       - update content in file /opt/gitlab/sv/registry/run from none to acccae
ESC[36mweb_1  |ESC[0m       --- /opt/gitlab/sv/registry/run     2023-03-05 03:47:54.597720561 +0000
ESC[36mweb_1  |ESC[0m       +++ /opt/gitlab/sv/registry/.chef-run20230305-30-eiqcdn     2023-03-05 03:47:54.597720561 +0000
ESC[36mweb_1  |ESC[0m       +cd /var/opt/gitlab/registry
ESC[36mweb_1  |ESC[0m       +exec chpst -e /opt/gitlab/etc/registry/env -P \

ESC[36mweb_1  |ESC[0m       +  -U registry:registry \
ESC[36mweb_1  |ESC[0m       +  -u registry:registry \
ESC[36mweb_1  |ESC[0m       +  /opt/gitlab/embedded/bin/registry serve ./config.yml[2023-03-05T03:47:54+00:00] INFO: template[/opt/gitlab/sv/registry/run] owner changed to 0
ESC[36mweb_1  |ESC[0m [2023-03-05T03:47:54+00:00] INFO: template[/opt/gitlab/sv/registry/run] group changed to 0
ESC[36mweb_1  |ESC[0m [2023-03-05T03:47:54+00:00] INFO: template[/opt/gitlab/sv/registry/run] mode changed to 755
ESC[36mweb_1  |ESC[0m     * directory[/opt/gitlab/sv/registry/log] action create[2023-03-05T03:47:54+00:00] INFO: directory[/opt/gitlab/sv/registry/log] created directory /opt/gitlab/sv/registry/log
ESC[36mweb_1  |ESC[0m       - create new directory /opt/gitlab/sv/registry/log[2023-03-05T03:47:54+00:00] INFO: directory[/opt/gitlab/sv/registry/log] owner changed to 0
ESC[36mweb_1  |ESC[0m [2023-03-05T03:47:54+00:00] INFO: directory[/opt/gitlab/sv/registry/log] group changed to 0
ESC[36mweb_1  |ESC[0m [2023-03-05T03:47:54+00:00] INFO: directory[/opt/gitlab/sv/registry/log] mode changed to 755
ESC[36mweb_1  |ESC[0m     * directory[/opt/gitlab/sv/registry/log/main] action create[2023-03-05T03:47:54+00:00] INFO: directory[/opt/gitlab/sv/registry/log/main] created directory /opt/gitlab/sv/registry/log/main
ESC[36mweb_1  |ESC[0m       - create new directory /opt/gitlab/sv/registry/log/main[2023-03-05T03:47:54+00:00] INFO: directory[/opt/gitlab/sv/registry/log/main] owner changed to 0
ESC[36mweb_1  |ESC[0m [2023-03-05T03:47:54+00:00] INFO: directory[/opt/gitlab/sv/registry/log/main] group changed to 0
ESC[36mweb_1  |ESC[0m [2023-03-05T03:47:54+00:00] INFO: directory[/opt/gitlab/sv/registry/log/main] mode changed to 755
ESC[36mweb_1  |ESC[0m     * template[/opt/gitlab/sv/registry/log/config] action create[2023-03-05T03:47:54+00:00] INFO: template[/opt/gitlab/sv/registry/log/config] created file /opt/gitlab/sv/registry/log/config
ESC[36mweb_1  |ESC[0m       - create new file /opt/gitlab/sv/registry/log/config[2023-03-05T03:47:54+00:00] INFO: template[/opt/gitlab/sv/registry/log/config] updated file contents /opt/gitlab/sv/registry/log/config
ESC[36mweb_1  |ESC[0m       - update content in file /opt/gitlab/sv/registry/log/config from none to 623c00
ESC[36mweb_1  |ESC[0m       --- /opt/gitlab/sv/registry/log/config      2023-03-05 03:47:54.633720989 +0000
ESC[36mweb_1  |ESC[0m       +++ /opt/gitlab/sv/registry/log/.chef-config20230305-30-1cq5tzd     2023-03-05 03:47:54.633720989 +0000
ESC[36mweb_1  |ESC[0m       +[2023-03-05T03:47:54+00:00] INFO: template[/opt/gitlab/sv/registry/log/config] owner changed to 0
ESC[36mweb_1  |ESC[0m [2023-03-05T03:47:54+00:00] INFO: template[/opt/gitlab/sv/registry/log/config] group changed to 0
ESC[36mweb_1  |ESC[0m [2023-03-05T03:47:54+00:00] INFO: template[/opt/gitlab/sv/registry/log/config] mode changed to 644
ESC[36mweb_1  |ESC[0m [2023-03-05T03:47:54+00:00] INFO: template[/opt/gitlab/sv/registry/log/config] sending create action to ruby_block[verify_chown_persisted_on_registry] (immediate)
ESC[36mweb_1  |ESC[0m     * ruby_block[verify_chown_persisted_on_registry] action create[2023-03-05T03:47:54+00:00] INFO: ruby_block[verify_chown_persisted_on_registry] called
ESC[36mweb_1  |ESC[0m       - execute the ruby block verify_chown_persisted_on_registry
ESC[36mweb_1  |ESC[0m     * ruby_block[verify_chown_persisted_on_registry] action nothing (skipped due to action :nothing)
ESC[36mweb_1  |ESC[0m     * link[/var/log/gitlab/registry/config] action create[2023-03-05T03:47:54+00:00] INFO: link[/var/log/gitlab/registry/config] created
ESC[36mweb_1  |ESC[0m       - create symlink at /var/log/gitlab/registry/config to /opt/gitlab/sv/registry/log/config
ESC[36mweb_1  |ESC[0m     * template[/opt/gitlab/sv/registry/log/run] action create[2023-03-05T03:47:54+00:00] INFO: template[/opt/gitlab/sv/registry/log/run] created file /opt/gitlab/sv/registry/log/run
ESC[36mweb_1  |ESC[0m       - create new file /opt/gitlab/sv/registry/log/run[2023-03-05T03:47:54+00:00] INFO: template[/opt/gitlab/sv/registry/log/run] updated file contents /opt/gitlab/sv/registry/log/run
ESC[36mweb_1  |ESC[0m       - update content in file /opt/gitlab/sv/registry/log/run from none to cb6440
ESC[36mweb_1  |ESC[0m       --- /opt/gitlab/sv/registry/log/run 2023-03-05 03:47:54.673721463 +0000
ESC[36mweb_1  |ESC[0m       +++ /opt/gitlab/sv/registry/log/.chef-run20230305-30-1uoib66        2023-03-05 03:47:54.673721463 +0000
ESC[36mweb_1  |ESC[0m       +exec svlogd -tt /var/log/gitlab/registry[2023-03-05T03:47:54+00:00] INFO: template[/opt/gitlab/sv/registry/log/run] owner changed to 0
ESC[36mweb_1  |ESC[0m [2023-03-05T03:47:54+00:00] INFO: template[/opt/gitlab/sv/registry/log/run] group changed to 0
ESC[36mweb_1  |ESC[0m [2023-03-05T03:47:54+00:00] INFO: template[/opt/gitlab/sv/registry/log/run] mode changed to 755
ESC[36mweb_1  |ESC[0m     * directory[/opt/gitlab/sv/registry/env] action create[2023-03-05T03:47:54+00:00] INFO: directory[/opt/gitlab/sv/registry/env] created directory /opt/gitlab/sv/registry/env
ESC[36mweb_1  |ESC[0m       - create new directory /opt/gitlab/sv/registry/env[2023-03-05T03:47:54+00:00] INFO: directory[/opt/gitlab/sv/registry/env] owner changed to 0
ESC[36mweb_1  |ESC[0m [2023-03-05T03:47:54+00:00] INFO: directory[/opt/gitlab/sv/registry/env] group changed to 0
ESC[36mweb_1  |ESC[0m [2023-03-05T03:47:54+00:00] INFO: directory[/opt/gitlab/sv/registry/env] mode changed to 755
ESC[36mweb_1  |ESC[0m     * ruby_block[Delete unmanaged env files for registry service] action run (skipped due to only_if)
ESC[36mweb_1  |ESC[0m     * template[/opt/gitlab/sv/registry/check] action create (skipped due to only_if)
ESC[36mweb_1  |ESC[0m     * template[/opt/gitlab/sv/registry/finish] action create (skipped due to only_if)
ESC[36mweb_1  |ESC[0m     * directory[/opt/gitlab/sv/registry/control] action create[2023-03-05T03:47:54+00:00] INFO: directory[/opt/gitlab/sv/registry/control] created directory /opt/gitlab/sv/registry/control
ESC[36mweb_1  |ESC[0m       - create new directory /opt/gitlab/sv/registry/control[2023-03-05T03:47:54+00:00] INFO: directory[/opt/gitlab/sv/registry/control] owner changed to 0
ESC[36mweb_1  |ESC[0m [2023-03-05T03:47:54+00:00] INFO: directory[/opt/gitlab/sv/registry/control] group changed to 0
ESC[36mweb_1  |ESC[0m [2023-03-05T03:47:54+00:00] INFO: directory[/opt/gitlab/sv/registry/control] mode changed to 755
ESC[36mweb_1  |ESC[0m     * link[/opt/gitlab/init/registry] action create[2023-03-05T03:47:54+00:00] INFO: link[/opt/gitlab/init/registry] created
ESC[36mweb_1  |ESC[0m       - create symlink at /opt/gitlab/init/registry to /opt/gitlab/embedded/bin/sv
ESC[36mweb_1  |ESC[0m     * file[/opt/gitlab/sv/registry/down] action nothing (skipped due to action :nothing)
ESC[36mweb_1  |ESC[0m [2023-03-05T03:47:54+00:00] INFO: template[/opt/gitlab/sv/registry/run] sending run action to ruby_block[restart_service] (delayed)
ESC[36mweb_1  |ESC[0m [2023-03-05T03:47:54+00:00] INFO: template[/opt/gitlab/sv/registry/log/config] sending create action to ruby_block[reload_log_service] (delayed)
ESC[36mweb_1  |ESC[0m       * directory[/opt/gitlab/sv/registry] action create (up to date)
ESC[36mweb_1  |ESC[0m       * template[/opt/gitlab/sv/registry/run] action create (up to date)
ESC[36mweb_1  |ESC[0m       * directory[/opt/gitlab/sv/registry/log] action create (up to date)
ESC[36mweb_1  |ESC[0m       * directory[/opt/gitlab/sv/registry/log/main] action create (up to date)
ESC[36mweb_1  |ESC[0m       * template[/opt/gitlab/sv/registry/log/config] action create (up to date)
ESC[36mweb_1  |ESC[0m       * ruby_block[verify_chown_persisted_on_registry] action nothing (skipped due to action :nothing)
ESC[36mweb_1  |ESC[0m       * link[/var/log/gitlab/registry/config] action create (up to date)
ESC[36mweb_1  |ESC[0m       * template[/opt/gitlab/sv/registry/log/run] action create (up to date)
ESC[36mweb_1  |ESC[0m       * directory[/opt/gitlab/sv/registry/env] action create (up to date)
ESC[36mweb_1  |ESC[0m       * ruby_block[Delete unmanaged env files for registry service] action run (skipped due to only_if)
ESC[36mweb_1  |ESC[0m       * template[/opt/gitlab/sv/registry/check] action create (skipped due to only_if)
ESC[36mweb_1  |ESC[0m       * template[/opt/gitlab/sv/registry/finish] action create (skipped due to only_if)
ESC[36mweb_1  |ESC[0m       * directory[/opt/gitlab/sv/registry/control] action create (up to date)
ESC[36mweb_1  |ESC[0m       * link[/opt/gitlab/init/registry] action create (up to date)
ESC[36mweb_1  |ESC[0m       * file[/opt/gitlab/sv/registry/down] action nothing (skipped due to action :nothing)
ESC[36mweb_1  |ESC[0m       * link[/opt/gitlab/service/registry] action create[2023-03-05T03:47:54+00:00] INFO: link[/opt/gitlab/service/registry] created
ESC[36mweb_1  |ESC[0m         - create symlink at /opt/gitlab/service/registry to /opt/gitlab/sv/registry
ESC[36mweb_1  |ESC[0m       * ruby_block[wait for registry service socket] action run[2023-03-05T03:47:57+00:00] INFO: ruby_block[wait for registry service socket] called
ESC[36mweb_1  |ESC[0m         - execute the ruby block wait for registry service socket
ESC[36mweb_1  |ESC[0m [2023-03-05T03:47:57+00:00] INFO: template[/opt/gitlab/sv/registry/log/run] sending run action to ruby_block[restart_log_service] (delayed)
ESC[36mweb_1  |ESC[0m       * directory[/opt/gitlab/sv/registry] action create (up to date)
ESC[36mweb_1  |ESC[0m       * template[/opt/gitlab/sv/registry/run] action create (up to date)
ESC[36mweb_1  |ESC[0m       * directory[/opt/gitlab/sv/registry/log] action create (up to date)
ESC[36mweb_1  |ESC[0m       * directory[/opt/gitlab/sv/registry/log/main] action create (up to date)
ESC[36mweb_1  |ESC[0m       * template[/opt/gitlab/sv/registry/log/config] action create (up to date)
ESC[36mweb_1  |ESC[0m       * ruby_block[verify_chown_persisted_on_registry] action nothing (skipped due to action :nothing)
ESC[36mweb_1  |ESC[0m       * link[/var/log/gitlab/registry/config] action create (up to date)
ESC[36mweb_1  |ESC[0m       * template[/opt/gitlab/sv/registry/log/run] action create (up to date)
ESC[36mweb_1  |ESC[0m       * directory[/opt/gitlab/sv/registry/env] action create (up to date)
ESC[36mweb_1  |ESC[0m       * ruby_block[Delete unmanaged env files for registry service] action run (skipped due to only_if)
ESC[36mweb_1  |ESC[0m       * template[/opt/gitlab/sv/registry/check] action create (skipped due to only_if)
ESC[36mweb_1  |ESC[0m       * template[/opt/gitlab/sv/registry/finish] action create (skipped due to only_if)
ESC[36mweb_1  |ESC[0m       * directory[/opt/gitlab/sv/registry/control] action create (up to date)
ESC[36mweb_1  |ESC[0m       * link[/opt/gitlab/init/registry] action create (up to date)
ESC[36mweb_1  |ESC[0m       * file[/opt/gitlab/sv/registry/down] action nothing (skipped due to action :nothing)
ESC[36mweb_1  |ESC[0m       * link[/opt/gitlab/service/registry] action create (up to date)
ESC[36mweb_1  |ESC[0m       * ruby_block[wait for registry service socket] action run (skipped due to not_if)
ESC[36mweb_1  |ESC[0m     * link[/opt/gitlab/service/registry] action create (up to date)
ESC[36mweb_1  |ESC[0m     * ruby_block[wait for registry service socket] action run (skipped due to not_if)
ESC[36mweb_1  |ESC[0m     * file[/var/opt/gitlab/registry/VERSION] action create[2023-03-05T03:48:08+00:00] INFO: file[/var/opt/gitlab/registry/VERSION] created file /var/opt

registry process is running:

root         374  0.0  0.0   2372   620 ?        Ss   15:09   0:00 runsv registry
root         376  0.0  0.0   2516   672 ?        S    15:09   0:00 svlogd -tt /var/log/gitlab/registry
registry     378  0.0  0.5 1423208 32232 ?       Ssl  15:09   0:01 /opt/gitlab/embedded/bin/registry serve ./config.yml
git          542 12.2 14.2 1480076 868756 ?      Sl   15:10   8:32 sidekiq 6.5.7 queues:authorized_project_update:authorized_project_update_project_recalculate,authorized_project_update:authorized_project_update_project_recalculate_per_user,authorized_project_update:authorized_project_update_user_refresh_from_replica,authorized_project_update:authorized_project_update_user_refresh_over_user_range,authorized_project_update:authorized_project_update_user_refresh_with_low_urgency,auto_devops:auto_devops_disable,auto_merge:auto_merge_process,batched_background_migrations:database_batched_background_migration_ci_execution,batched_background_migrations:database_batched_background_migration_main_execution,chaos:chaos_cpu_spin,chaos:chaos_db_spin,chaos:chaos_kill,chaos:chaos_leak_mem,chaos:chaos_sleep,cluster_agent:clusters_agents_delete_expired_events,container_repository:cleanup_container_repository,container_repository:container_expiration_policies_cleanup_container_repository,container_repository:delete_container_repository,container_repository_delete:container_registry_delete_
root         745  0.0  0.0   4296   488 ?        S    15:13   0:00 tail --follow=name --retry /var/log/gitlab/redis/current /var/log/gitlab/postgres-exporter/current /var/log/gitlab/grafana/current /var/log/gitlab/logrotate/current /var/log/gitlab/gitaly/current /var/log/gitlab/gitaly/gitaly_ruby_json.log /var/log/gitlab/nginx/gitlab_registry_error.log /var/log/gitlab/nginx/current /var/log/gitlab/nginx/gitlab_access.log /var/log/gitlab/nginx/error.log /var/log/gitlab/nginx/gitlab_registry_access.log /var/log/gitlab/nginx/access.log /var/log/gitlab/nginx/gitlab_error.log /var/log/gitlab/puma/current /var/log/gitlab/puma/puma_stdout.log /var/log/gitlab/puma/puma_stderr.log /var/log/gitlab/redis-exporter/current /var/log/gitlab/gitlab-kas/current /var/log/gitlab/sshd/current /var/log/gitlab/registry/current /var/log/gitlab/sidekiq/current /var/log/gitlab/gitlab-exporter/current /var/log/gitlab/postgresql/current /var/log/gitlab/gitlab-workhorse/current /var/log/gitlab/prometheus/current /var/log/gitlab/gitlab-rails/production_json.log /var/log/gitlab/gitlab-rails/auth.log /var/log/gitlab/gitlab-rails/application.log /var/log/gitlab/gitlab-rails/graphql_json.log /var/log/gitlab/gitlab-rails/exceptions_json.log /var/log/gitlab/gitlab-rails/application_json.log /var/log/gitlab/gitlab-rails/service_measurement.log /var/log/gitlab/gitlab-rails/database_load_balancing.log /var/log/gitlab/gitlab-rails/audit_json.log /var/log/gitlab/gitlab-rails/git_json.log /var/log/gitlab/gitlab-rails/production.log /var/log/gitlab/gitlab-rails/sidekiq_client.log /var/log/gitlab/gitlab-rails/api_json.log /var/log/gitlab/gitlab-rails/grpc.log /var/log/gitlab/alertmanager/current

what i need is a way to debugg this, since i cannot figure it out by myself.

Bodo · March 9, 2023, 7:27am

Hi Belutak,

i am facing the same problem. I have invested a lot of time but so far without a solution

Belutak · March 9, 2023, 10:55am

i will make sure to replay you here if i get to solution

iwalker · March 9, 2023, 11:00am

Did you expose the 5005 port with docker? If not, then perhaps easier way is have two hostnames, gitlab.mydomain.com and gitlab-registry.mydomain.com so that the nginx proxy can deal with which one is for registry and which one is for normal gitlab commits, etc. Then just access it over port 80 or 443 which you already have exposed.

Otherwise you need to expose the port 5005. You can check that with docker ps or docker container ls. I don’t use docker, but I expect you problem could be because of this, but you would need to provide more detail to be sure.

dnsmichi · March 9, 2023, 12:15pm

Please run

docker ps -a

to see the running container, and all of its port mappings. If 0.0.0.0:5005 is missing, this can be the next debug step to verify why docker-compose is not respecting the configuration.

If port 5005 is bound to the host external network, change the debug direction to external observability.

Run nmap to perform a port scan from an external host (the same which executes docker login)

nmap -A registry.gitlab2.arma

Connect to the TLS HTTPS endpoint using openssl s_client to see if the TLS handshake is successful (docker CLI hides some errors).

openssl s_client -connect registry.gitlab2.arma:5005

Please share the output of all commands in full. If the port is not visible from external, check the host firewall preventing access - iptables, ufw, or anything else. SELinux also comes to mind.

Belutak · March 9, 2023, 1:03pm

Thank you so much for answering! Am dealing with this since friday :S

docker ps -a

CONTAINER ID   IMAGE                     COMMAND             CREATED      STATUS                  PORTS                                                                                                             NAMES
94f168a8b9df   gitlab/gitlab-ce:latest   "/assets/wrapper"   4 days ago   Up 22 hours (healthy)   0.0.0.0:80->80/tcp, :::80->80/tcp, 0.0.0.0:443->443/tcp, :::443->443/tcp, 0.0.0.0:2243->22/tcp, :::2243->22/tcp   gitlab_web_1

#from host on which is gitlab
nmap -A registry.gitlab2.arma

Starting Nmap 7.80 ( https://nmap.org ) at 2023-03-09 13:45 CET
Nmap scan report for registry.gitlab2.arma (10.0.0.199)
Host is up (0.000040s latency).
rDNS record for 10.0.0.199: gitlab2
Not shown: 997 closed ports
PORT    STATE    SERVICE VERSION
22/tcp  open     ssh     OpenSSH 8.2p1 Ubuntu 4ubuntu0.5 (Ubuntu Linux; protocol 2.0)
80/tcp  filtered http
443/tcp filtered https
Device type: general purpose
Running: Linux 2.6.X
OS CPE: cpe:/o:linux:linux_kernel:2.6.32
OS details: Linux 2.6.32
Network Distance: 0 hops
Service Info: OS: Linux; CPE: cpe:/o:linux:linux_kernel

OS and Service detection performed. Please report any incorrect results at https://nmap.org/submit/ .
Nmap done: 1 IP address (1 host up) scanned in 8.24 seconds

same from different machine on same network segment:

Starting Nmap 7.60 ( https://nmap.org ) at 2023-03-09 13:44 CET
Nmap scan report for registry.gitlab2.arma (10.0.0.199)
Host is up (0.0091s latency).
rDNS record for 10.0.0.199: gitlab2
Not shown: 997 closed ports
PORT    STATE SERVICE  VERSION
22/tcp  open  ssh      OpenSSH 8.2p1 Ubuntu 4ubuntu0.5 (Ubuntu Linux; protocol 2.0)
80/tcp  open  http     nginx
| http-robots.txt: 57 disallowed entries (15 shown)
| / /autocomplete/users /autocomplete/projects /search 
| /admin /profile /dashboard /users /api/v* /help /s/ /-/profile 
|_/-/ide/ /-/experiment /*/new
|_http-server-header: nginx
|_http-title: Did not follow redirect to https://registry.gitlab2.arma/users/sign_in
443/tcp open  ssl/http nginx
| http-robots.txt: 57 disallowed entries (15 shown)
| / /autocomplete/users /autocomplete/projects /search 
| /admin /profile /dashboard /users /api/v* /help /s/ /-/profile 
|_/-/ide/ /-/experiment /*/new
|_http-server-header: nginx
| http-title: Sign in \xC2\xB7 GitLab
|_Requested resource was https://registry.gitlab2.arma/users/sign_in
| ssl-cert: Subject: commonName=Digi/organizationName=TI/stateOrProvinceName=Serbia/countryName=RS
| Not valid before: 2023-03-05T00:26:09
|_Not valid after:  2024-03-04T00:26:09
MAC Address: 00:0C:29:28:19:37 (VMware)
No exact OS matches for host (If you know what OS is running on it, see https://nmap.org/submit/ ).
TCP/IP fingerprint:
OS:SCAN(V=7.60%E=4%D=3/9%OT=22%CT=1%CU=36184%PV=Y%DS=1%DC=D%G=Y%M=000C29%TM
OS:=6409D4C9%P=x86_64-pc-linux-gnu)SEQ(SP=104%GCD=1%ISR=10D%TI=Z%CI=Z%TS=A)
OS:SEQ(SP=104%GCD=1%ISR=10D%TI=Z%CI=Z%II=I%TS=A)OPS(O1=M5B4ST11NW7%O2=M5B4S
OS:T11NW7%O3=M5B4NNT11NW7%O4=M5B4ST11NW7%O5=M5B4ST11NW7%O6=M5B4ST11)WIN(W1=
OS:FE88%W2=FE88%W3=FE88%W4=FE88%W5=FE88%W6=FE88)ECN(R=Y%DF=Y%T=40%W=FAF0%O=
OS:M5B4NNSNW7%CC=Y%Q=)T1(R=Y%DF=Y%T=40%S=O%A=S+%F=AS%RD=0%Q=)T2(R=N)T3(R=N)
OS:T4(R=Y%DF=Y%T=40%W=0%S=A%A=Z%F=R%O=%RD=0%Q=)T5(R=Y%DF=Y%T=40%W=0%S=Z%A=S
OS:+%F=AR%O=%RD=0%Q=)T6(R=Y%DF=Y%T=40%W=0%S=A%A=Z%F=R%O=%RD=0%Q=)T7(R=Y%DF=
OS:Y%T=40%W=0%S=Z%A=S+%F=AR%O=%RD=0%Q=)U1(R=Y%DF=N%T=40%IPL=164%UN=0%RIPL=G
OS:%RID=G%RIPCK=G%RUCK=G%RUD=G)IE(R=Y%DFI=N%T=40%CD=S)

Network Distance: 1 hop
Service Info: OS: Linux; CPE: cpe:/o:linux:linux_kernel

TRACEROUTE
HOP RTT     ADDRESS
1   9.10 ms gitlab2 (10.0.0.199)

OS and Service detection performed. Please report any incorrect results at https://nmap.org/submit/ .
Nmap done: 1 IP address (1 host up) scanned in 40.76 seconds

openssl s_client -connect registry.gitlab2.arma:5005

140554093524288:error:0200206F:system library:connect:Connection refused:../crypto/bio/b_sock2.c:110:
140554093524288:error:2008A067:BIO routines:BIO_connect:connect error:../crypto/bio/b_sock2.c:111:
connect:errno=111

ufw is disabled, i did not touch ip tables, apparmor is default and enabled, OS is ubuntu server 20.04

the problem is that registry never starts to listen on port 5005, without any error that i can find in logs.

If port 5005 is bound to the host external network, change the debug direction to external observability

i want us to understand each other completely and this i dont understand 100%. There is no need for registry to be really open for external access(from outside of scope of my local network), but i want it to be accessible from my local network, just like gitlab instance is. My docker-compose file is dealing with that and i think thats ok:

restart: always
  hostname: 'gitlab2.arma'

  environment:
    GITLAB_OMNIBUS_CONFIG: |
      # Add any other gitlab.rb configuration here, each on its own line
      external_url 'https://gitlab2.arma'
      gitlab_rails['gitlab_shell_ssh_port'] = 22
      nginx['redirect_http_to_https'] = false
      nginx['ssl_certificate'] = "/etc/gitlab/ssl/ca.crt"
      nginx['ssl_certificate_key'] = "/etc/gitlab/ssl/ca.key"
      nginx['ssl_dhparam'] = "/etc/gitlab/ssl/dhparams.pem"  

  ports:
    - '443:443'
    - '80:80'
    - '2243:22'
    - '5005:5005'
  volumes:
    - '${GITLAB_HOME}/config:/etc/gitlab'
    - '${GITLAB_HOME}/logs:/var/log/gitlab'
    - '${GITLAB_HOME}/data:/var/opt/gitlab'
    - '${GITLAB_HOME}/config/ssl:/etc/gitlab/ssl'

so i think that problem is in gitlab container itself, that is for some reason, not starting to listen on port 5005

heres nginx conf file that is made inside of container for registry

# This file is managed by gitlab-ctl. Manual changes will be
# erased! To change the contents below, edit /etc/gitlab/gitlab.rb
# and run `sudo gitlab-ctl reconfigure`.

## Lines starting with two hashes (##) are comments with information.
## Lines starting with one hash (#) are configuration parameters that can be uncommented.
##
###################################
##         configuration         ##
###################################


server { ## HTTPS server
  listen *:5005 ssl;
  server_name registry.gitlab2.arma;
  server_tokens off; ## Don't show the nginx version number, a security best practice

  client_max_body_size 0;
  chunked_transfer_encoding on;

  ## Strong SSL Security
  ## https://raymii.org/s/tutorials/Strong_SSL_Security_On_nginx.html & https://cipherli.st/
  ssl_certificate /etc/gitlab/ssl/ca.crt;
  ssl_certificate_key /etc/gitlab/ssl/ca.key;

  ssl_ciphers 'ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384';
  ssl_protocols  TLSv1.2 TLSv1.3;
  ssl_prefer_server_ciphers off;
  ssl_session_cache  shared:SSL:10m;
  ssl_session_tickets off;
  ssl_session_timeout  1d;



  ## Real IP Module Config
  ## http://nginx.org/en/docs/http/ngx_http_realip_module.html

  ## HSTS Config
  ## https://www.nginx.com/blog/http-strict-transport-security-hsts-and-nginx/
  add_header Strict-Transport-Security "max-age=63072000";

  access_log  /var/log/gitlab/nginx/gitlab_registry_access.log gitlab_access;
  error_log   /var/log/gitlab/nginx/gitlab_registry_error.log error;


  location / {

    proxy_set_header Host $http_host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto https;
    proxy_set_header X-Forwarded-Ssl on;

    proxy_read_timeout                  900;
    proxy_cache off;
    proxy_buffering off;
    proxy_request_buffering off;
    proxy_http_version 1.1;

    proxy_pass          http://localhost:5000;
  }

  
} ## end HTTPS server

Belutak · March 9, 2023, 1:08pm

thank you iwalker!

did you check my docker-compose file? I do expose the ports


web:
  image: 'gitlab/gitlab-ce:latest'
  restart: always
  hostname: 'gitlab2.arma'

  environment:
    GITLAB_OMNIBUS_CONFIG: |
      # Add any other gitlab.rb configuration here, each on its own line
      external_url 'https://gitlab2.arma'
      gitlab_rails['gitlab_shell_ssh_port'] = 22
      nginx['redirect_http_to_https'] = false
      nginx['ssl_certificate'] = "/etc/gitlab/ssl/ca.crt"
      nginx['ssl_certificate_key'] = "/etc/gitlab/ssl/ca.key"
      nginx['ssl_dhparam'] = "/etc/gitlab/ssl/dhparams.pem"  

  ports:
    - '443:443'
    - '80:80'
    - '2243:22'
    - '5005:5005'
  volumes:
    - '${GITLAB_HOME}/config:/etc/gitlab'
    - '${GITLAB_HOME}/logs:/var/log/gitlab'
    - '${GITLAB_HOME}/data:/var/opt/gitlab'
    - '${GITLAB_HOME}/config/ssl:/etc/gitlab/ssl'

iwalker · March 9, 2023, 1:09pm

Hi,

As you can see despite your docker compose having port 5005, it’s not exposed…the docker ps command shows ports, but not 5005. So there is no way to communicate on this port until it’s opened.

dnsmichi · March 9, 2023, 1:26pm

Thanks for providing the outputs. openssl and nmap failed to detect the port because it is not exposed.

I assume that *.arma is your local domain namespace, with IP addresses and network filters that only allow hosts from your internal network to access that host and domain. By “external” I mean external to the GitLab host but local to the environment network.

I cannot reach gitlab2.arma from the internet.

Next steps for debugging

The output from docker ps -a

0.0.0.0:80->80/tcp, :::80->80/tcp, 0.0.0.0:443->443/tcp, :::443->443/tcp, 0.0.0.0:2243->22/tcp, :::2243->22/tcp

provides the next debugging steps

Either the registry service is not started to listen on port 5005
Or docker does not expose the port correctly.

I’d suggest attaching to the running container using docker exec and check things inside the container.

# get the container id
docker ps -a 

docker exec -ti 94f168a8b9df bash

and follow the troubleshooting docs to check the logs manually inside the container.

One other idea I had while reading the docs - enable the Registry debug server from the docker-compose config, restart docker-compose, exec into the running container again, and run the curl commands to request the debug output from the server.

Search for errors in the log, and verify to redact sensitive information before posting here.

Another idea - maybe port 5005 is already assigned by a stale process/socket, and the registry bails out on start. This should be logged in /var/log/gitlab/registry/current - ${GITLAB_HOME}/logs/registry/current

Try changing the port to 6000 and see if the behaviour changes. If it starts to work - restart Docker, and the host VM to clear out stale sockets. Investigate the startup routines if there are services that bind port 5005.

Belutak · March 9, 2023, 3:41pm

there are some changes, it seams that you were right about stale sockets i think, because there are changes

docker ps -a

CONTAINER ID   IMAGE                     COMMAND             CREATED             STATUS                       PORTS                                                                                                                                                                                                   NAMES
066a7129d09b   gitlab/gitlab-ce:latest   "/assets/wrapper"   About an hour ago   Up About an hour (healthy)   0.0.0.0:80->80/tcp, :::80->80/tcp, 0.0.0.0:443->443/tcp, :::443->443/tcp, 0.0.0.0:5005->5005/tcp, :::5005->5005/tcp, 0.0.0.0:6006->6006/tcp, :::6006->6006/tcp, 0.0.0.0:2243->22/tcp, :::2243->22/tcp   gitlab_web_1

inside of docker container:
netstat -tn

Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       
tcp        0      0 127.0.0.1:50276         127.0.0.1:9168          ESTABLISHED 
tcp        0      0 127.0.0.1:8080          127.0.0.1:46602         TIME_WAIT   
tcp        0      0 127.0.0.1:46724         127.0.0.1:9090          ESTABLISHED 
tcp        0      0 127.0.0.1:8080          127.0.0.1:36508         TIME_WAIT   
tcp        0      0 127.0.0.1:59346         127.0.0.1:9187          ESTABLISHED 
tcp        0      0 127.0.0.1:50288         127.0.0.1:9168          ESTABLISHED 
tcp        0      0 127.0.0.1:47726         127.0.0.1:8060          ESTABLISHED 
tcp        0      0 127.0.0.1:8060          127.0.0.1:47726         ESTABLISHED 
tcp        0      0 127.0.0.1:47372         127.0.0.1:9229          ESTABLISHED 
tcp        0      0 127.0.0.1:51372         127.0.0.1:9121          ESTABLISHED 
tcp        0      0 127.0.0.1:8080          127.0.0.1:35052         TIME_WAIT   
tcp        0      0 172.17.0.2:80           10.0.0.195:60486        ESTABLISHED 
tcp        0      0 127.0.0.1:55738         127.0.0.1:443           TIME_WAIT   
tcp        0      0 127.0.0.1:8080          127.0.0.1:32848         TIME_WAIT   
tcp        0      0 127.0.0.1:58152         127.0.0.1:9236          ESTABLISHED 
tcp        0      0 127.0.0.1:9229          127.0.0.1:47372         ESTABLISHED 
tcp        0      0 127.0.0.1:443           127.0.0.1:55738         TIME_WAIT   
tcp        0      0 127.0.0.1:9168          127.0.0.1:50288         ESTABLISHED 
tcp        0      0 127.0.0.1:37954         127.0.0.1:9168          ESTABLISHED 
tcp        0      0 127.0.0.1:9090          127.0.0.1:46724         ESTABLISHED 
tcp        0      0 127.0.0.1:9236          127.0.0.1:58152         ESTABLISHED 
tcp        0      0 127.0.0.1:9121          127.0.0.1:51372         ESTABLISHED 
tcp        0      0 127.0.0.1:9168          127.0.0.1:50276         ESTABLISHED 
tcp        0      0 127.0.0.1:33928         127.0.0.1:8082          ESTABLISHED 
tcp        0      0 127.0.0.1:9168          127.0.0.1:37954         ESTABLISHED 
tcp        0      0 127.0.0.1:9187          127.0.0.1:59346         ESTABLISHED 
tcp        0      0 172.17.0.2:80           10.0.0.182:39096        ESTABLISHED 
tcp        0      0 127.0.0.1:8082          127.0.0.1:33928         ESTABLISHED 
tcp        0      0 127.0.0.1:8080          127.0.0.1:34906         TIME_WAIT

on host:
netstat -tnlp

Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 0.0.0.0:2243            0.0.0.0:*               LISTEN      1201344/docker-prox 
tcp        0      0 0.0.0.0:5005            0.0.0.0:*               LISTEN      1201323/docker-prox 
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      1201388/docker-prox 
tcp        0      0 127.0.0.53:53           0.0.0.0:*               LISTEN      1115381/systemd-res 
tcp        0      0 0.0.0.0:6006            0.0.0.0:*               LISTEN      1201301/docker-prox 
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      983/sshd: /usr/sbin 
tcp        0      0 0.0.0.0:443             0.0.0.0:*               LISTEN      1201366/docker-prox 
tcp6       0      0 :::2243                 :::*                    LISTEN      1201352/docker-prox 
tcp6       0      0 :::5005                 :::*                    LISTEN      1201330/docker-prox 
tcp6       0      0 :::80                   :::*                    LISTEN      1201395/docker-prox 
tcp6       0      0 :::6006                 :::*                    LISTEN      1201308/docker-prox 
tcp6       0      0 :::22                   :::*                    LISTEN      983/sshd: /usr/sbin 
tcp6       0      0 :::443                  :::*                    LISTEN      1201374/docker-prox

so there are new ports on host, and in docker ps but when i enter gitlab container there is no ports.

if you noticing something bad in conf please correct me if you can. I will try debug steps u sent later. thank you!

# gitlab_rails['gitlab_default_projects_features_container_registry'] = true
#    "SKIP" => "db,uploads,repositories,builds,artifacts,lfs,registry,pages"
##! Docs: https://docs.gitlab.com/ee/administration/container_registry.html
 registry_external_url 'https://registry.gitlab2.arma:6006'
 gitlab_rails['registry_enabled'] = true
 gitlab_rails['registry_host'] = "registry.gitlab2.arma"
 gitlab_rails['registry_port'] = "6006"
 gitlab_rails['registry_path'] = "/var/opt/gitlab/gitlab-rails/shared/registry"
# gitlab_rails['registry_notification_secret'] = nil
 gitlab_rails['registry_api_url'] = "http://localhost:5000"
 gitlab_rails['registry_key_path'] = "/etc/gitlab/ssl/ssl_registry.key"
# gitlab_rails['registry_issuer'] = "omnibus-gitlab-issuer"
 registry['enable'] = true
 registry['token_realm'] = "https://gitlab2.arma:443"
# registry['username'] = "registry"
# registry['group'] = "registry"
# registry['uid'] = nil
# registry['gid'] = nil
# registry['dir'] = "/var/opt/gitlab/registry"
 registry['registry_http_addr'] = "localhost:5000"
# registry['debug_addr'] = "localhost:5001"
 registry['log_directory'] = "/var/log/gitlab/registry"
 registry['env_directory'] = "/opt/gitlab/etc/registry/env"
 registry['env'] = {
 registry['log_level'] = "info"
# registry['log_formatter'] = "text"
 registry['rootcertbundle'] = "/etc/gitlab/ssl/ssl_registry.crt"
# registry['health_storagedriver_enabled'] = true
# registry['middleware'] = nil
# registry['storage_delete_enabled'] = true
# registry['validation_enabled'] = false
# registry['autoredirect'] = false
# registry['compatibility_schema1_enabled'] = false
###! Docs: https://docs.gitlab.com/ee/administration/packages/container_registry.html#configure-storage-for-the-container-registry
# registry['storage'] = {
# registry['notifications'] = [
### Default registry notifications
# registry['default_notifications_timeout'] = "500ms"
# registry['default_notifications_threshold'] = 5
# registry['default_notifications_backoff'] = "1s"
# registry['default_notifications_headers'] = {}
# this "Registry NGINX" section, using the key `registry_nginx`.  However, those
# `registry_nginx['some_setting']` and should be set separately.
 registry_nginx['enable'] = true
# registry_nginx['proxy_set_headers'] = {
# When the registry is automatically enabled using the same domain as `external_url`,
 registry_nginx['listen_port'] = 5006
 registry_nginx['ssl_certificate'] = "/etc/gitlab/ssl/ca.crt"
 registry_nginx['ssl_certificate_key'] = "/etc/gitlab/ssl/ca.key"
# gitlab_rails['geo_secondary_registry_consistency_worker'] = "* * * * *"
# gitlab_rails['geo_registry_replication_enabled'] = true
# gitlab_rails['geo_registry_replication_primary_api_url'] = 'https://example.com:5050'

omgold · March 15, 2023, 7:28am

I have the same problem, but I’m not running running Gitlab inside Docker, but what is called ‘Installation from source’ in the manual (I’m using the distro packages from Arch to be exact).

The registry definition in gitlab.yml is this:

  registry:
    enabled: true
    host: gitlab.***redacted***
    port: 5050
    # internal address to the registry, will be used by GitLab to directly communicate with API
    api_url: http://localhost:5000/
    key: /etc/httpd/secure/gitlab-registry.ecdsa.key.pem
    path: /srv/***redacted***/gitlab/shared/registry
    issuer: gitlab-***redacted***

Which should be more or less equivalent to what the OP does. So looking at Docker to find the issue might be the wrong track, I guess.

dnsmichi · March 21, 2023, 10:37am

Thanks for sharing. In this case on the host level, I’d suggest changing the debugging strategy to looking for open ports and bindings using the following commands:

netstat -an | grep 5005
netstat -tulpen | grep 5005

lsof -i tcp | grep 5005

ps aux | grep gitlab

I’m not familiar with ArchLinux packages and their paths, maybe there is a permission issue with folders and socket binding, or firewall in this case.

I had a peek into GitLab - ArchWiki but there is no mention of the registry, except for the config options. The source install uses a different location with gitlab.yml, than the Omnibus package installed in the Docker container. Looking at the options to configure the Puma webserver differently, a socket problem is a good first strategy to debug imho.

omgold · March 21, 2023, 10:58am

Actually that is what I already have been doing

Nothing is listening on the configured port. The first 3 commands come up empty.

The last one gives this (which looks fine to me):

gitlab    233676  0.0  0.0  18672  6848 ?        Ss   Mar06   0:23 /usr/lib/systemd/systemd --user
gitlab    233687  0.0  0.0 171284  4572 ?        S    Mar06   0:00 (sd-pam)
gitlab    470894  0.2  0.9 2192112 159340 ?      Ssl  Mar13  27:06 /usr/bin/gitaly /etc/gitlab-gitaly/config.toml
gitlab    470909  0.1  0.5 2710144 93220 ?       Sl   Mar13  13:53 ruby-2.7 /usr/share/webapps/gitlab-gitaly/ruby/bin/gitaly-ruby 470894 /tmp/gitaly-1822952599/sock.d/ruby.0
gitlab    470910  0.1  0.7 2783916 123844 ?      Sl   Mar13  14:17 ruby-2.7 /usr/share/webapps/gitlab-gitaly/ruby/bin/gitaly-ruby 470894 /tmp/gitaly-1822952599/sock.d/ruby.1
gitlab-+  527844  0.0  0.1 740480 30000 ?        Ssl  Mar14   1:31 /bin/registry serve /etc/docker/registry/config.yml
gitlab   1148268 15.2  5.0 1124336 831652 ?      Ssl  11:51   0:52 puma 5.6.5 (unix:///run/gitlab/gitlab.socket) [gitlab-puma-worker]
gitlab   1148269  0.5  0.3 112584 55076 ?        Ss   11:51   0:01 ruby-2.7 /usr/share/webapps/gitlab/vendor/bundle/ruby/2.7.0/bin/mail_room -q -c /usr/share/webapps/gitlab/config/mail_room.yml
gitlab   1148270 20.8  5.8 2030004 959740 ?      Ssl  11:51   1:11 sidekiq 6.5.7 gitlab [0 of 10 busy]
gitlab   1148271  0.2  0.2 1483960 48128 ?       Ssl  11:51   0:00 /usr/bin/gitlab-workhorse -listenUmask 0 -listenNetwork unix -listenAddr /run/gitlab/gitlab-workhorse.socket -authBackend http://localhost:8080 -authSocket /run/gitlab/gitlab.socket -documentRoot /usr/share/webapps/gitlab/public
gitlab   1148402  2.7  5.6 2232896 925588 ?      Sl   11:52   0:07 puma: cluster worker 0: 1148268 [gitlab-puma-worker]
gitlab   1148405  2.4  5.6 2162796 921220 ?      Sl   11:52   0:07 puma: cluster worker 1: 1148268 [gitlab-puma-worker]
gitlab   1148408  2.6  5.6 2093156 920056 ?      Sl   11:52   0:07 puma: cluster worker 2: 1148268 [gitlab-puma-worker]
postgres 1148417  0.0  0.0 218740 15744 ?        Ss   11:52   0:00 postgres: gitlab gitlabhq_production ::1(47330) idle
postgres 1148426  0.0  0.0 218740 15744 ?        Ss   11:52   0:00 postgres: gitlab gitlabhq_production ::1(47336) idle
postgres 1148433  0.0  0.0 218740 15748 ?        Ss   11:52   0:00 postgres: gitlab gitlabhq_production ::1(47346) idle
postgres 1148434  0.0  0.2 225152 48048 ?        Ss   11:52   0:00 postgres: gitlab gitlabhq_production ::1(47354) idle
postgres 1148435  0.2  0.3 226976 52704 ?        Ss   11:52   0:00 postgres: gitlab gitlabhq_production ::1(47368) idle
postgres 1148437  0.0  0.2 222472 45260 ?        Ss   11:52   0:00 postgres: gitlab gitlabhq_production ::1(47376) idle
postgres 1148439  0.1  0.2 223968 47772 ?        Ss   11:52   0:00 postgres: gitlab gitlabhq_production ::1(47392) idle
postgres 1148450  0.0  0.2 222180 36464 ?        Ss   11:52   0:00 postgres: gitlab gitlabhq_production ::1(47406) idle
postgres 1148469  0.0  0.2 221392 34020 ?        Ss   11:52   0:00 postgres: gitlab gitlabhq_production ::1(47418) idle
postgres 1148471  0.1  0.2 225116 48632 ?        Ss   11:52   0:00 postgres: gitlab gitlabhq_production ::1(47432) idle
postgres 1148475  0.1  0.2 226196 46816 ?        Ss   11:52   0:00 postgres: gitlab gitlabhq_production ::1(47438) idle
postgres 1148486  0.0  0.0 218740 15772 ?        Ss   11:52   0:00 postgres: gitlab gitlabhq_production ::1(47454) idle
postgres 1148487  0.0  0.1 219588 19340 ?        Ss   11:52   0:00 postgres: gitlab gitlabhq_production ::1(47456) idle
postgres 1148488  0.0  0.1 221348 25992 ?        Ss   11:52   0:00 postgres: gitlab gitlabhq_production ::1(47466) idle
postgres 1148489  0.0  0.1 220196 32384 ?        Ss   11:52   0:00 postgres: gitlab gitlabhq_production ::1(47476) idle
postgres 1148490  0.0  0.1 220608 23764 ?        Ss   11:52   0:00 postgres: gitlab gitlabhq_production ::1(47480) idle
postgres 1148491  0.0  0.2 221580 33284 ?        Ss   11:52   0:00 postgres: gitlab gitlabhq_production ::1(47486) idle

Gitlab in general works as it should.

omgold · March 21, 2023, 11:08am

About permissions. Not sure what it could be. The logs are not providing any hints in that respect.

There is no LSM (like Apparmor) active, but gitlab-workhorse.service has ProtectSystem=full set.

Binding to TCP ports should still work, filesystem access is severely limited, including for creation on named unix sockets under most paths,

Are there directories workhorse needs to write to for the registry to work?

nik2208 · September 20, 2023, 3:03pm

here u can find how I’ve been able to make the registry work. it actually works, but there some issue trying to browse it from my gitlab registry section