404 Errors on Registry Push

Hi all,

I’m currently running into an issue with my repository access returning a 404 and I’m not sure why. I can do a docker login to docker.example.com (where example.com is not actually my domain…), but as soon as I push I am greeted with a 404 error by Gitlab:

image

I’ve attempted several different configurations and port mappings for the registry. Unfortunately It’s not clear to me what each of these settings do - so I’m fairly certain I have things misconfigured:

########### #####################################################################
## Container Registry settings
##! Docs: https://docs.gitlab.com/ee/administration/container_registry.html
################################################################################

 registry_external_url 'https://docker.example.com'

### Settings used by GitLab application
 gitlab_rails['registry_enabled'] = true
 gitlab_rails['registry_host'] = "localhost"
 gitlab_rails['registry_port'] = "5000"
 gitlab_rails['registry_path'] = "/var/opt/gitlab/gitlab-rails/shared/registry"

# Notification secret, it's used to authenticate notification requests to GitLab application
# You only need to change this when you use external Registry service, otherwise
# it will be taken directly from notification settings of your Registry
# gitlab_rails['registry_notification_secret'] = nil

###! **Do not change the following 3 settings unless you know what you are
###!   doing**
# gitlab_rails['registry_api_url'] = "http://localhost:5000"
# gitlab_rails['registry_key_path'] = "/var/opt/gitlab/gitlab-rails/certificate.key"
# gitlab_rails['registry_issuer'] = "omnibus-gitlab-issuer"

### Settings used by Registry application
 registry['enable'] = true
 registry['username'] = "registry"
 registry['group'] = "registry"
 registry['uid'] = nil
 registry['gid'] = nil
 registry['dir'] = "/var/opt/gitlab/registry"
 registry['registry_http_addr'] = "localhost:5000"
 registry['debug_addr'] = "localhost:5001"
 registry['log_directory'] = "/var/log/gitlab/registry"
 registry['env_directory'] = "/opt/gitlab/etc/registry/env"
 registry['env'] = {
#   'SSL_CERT_DIR' => "/opt/gitlab/embedded/ssl/certs/"
   "REGISTRY_HTTP_RELATIVEURLS" => true
 }
# registry['log_level'] = "info"
# registry['log_formatter'] = "text"
# registry['rootcertbundle'] = "/var/opt/gitlab/registry/certificate.crt"
# registry['health_storagedriver_enabled'] = true
# registry['storage_delete_enabled'] = true
# registry['validation_enabled'] = false
# registry['autoredirect'] = false
# registry['compatibility_schema1_enabled'] = false

### Registry backend storage
###! Docs: https://docs.gitlab.com/ee/administration/packages/container_registry.html#configure-storage-for-the-container-registry
# registry['storage'] = {
#   's3' => {
#     'accesskey' => 's3-access-key',
#     'secretkey' => 's3-secret-key-for-access-key',
#     'bucket' => 'your-s3-bucket',
#     'region' => 'your-s3-region',
#     'regionendpoint' => 'your-s3-regionendpoint'
#   },
#   'redirect' => {
#     'disable' => false
#   }
# }
### Registry notifications endpoints
# registry['notifications'] = [
#   {
#     'name' => 'test_endpoint',
#     'url' => 'https://gitlab.example.com/notify2',
#     'timeout' => '500ms',
#     'threshold' => 5,
#     'backoff' => '1s',
#     'headers' => {
#       "Authorization" => ["AUTHORIZATION_EXAMPLE_TOKEN"]
#     }
#   }
# ]
### Default registry notifications
# registry['default_notifications_timeout'] = "500ms"
# registry['default_notifications_threshold'] = 5
# registry['default_notifications_backoff'] = "1s"
# registry['default_notifications_headers'] = {}

Basically all I want is to be able to access the registry and push/pull. I don’t really care what port it is on as I will remap the port to 5000 with a reverse proxy and then translate it to the correct port when it hits the docker container local network

Oh, I’m using docker-compose to build and run the docker image:
version: " 3"

services:
  nginx-proxy:
    restart: always
    image: jwilder/nginx-proxy
    ports:
      - "80:80"
      - "443:443"
      - "5005:5005"
      - "5000:5000"
      - "5001:5001"
    volumes:
      - /var/run/docker.sock:/tmp/docker.sock:ro
      - /srv/nginx/certs:/etc/nginx/certs
  web:
    restart: always
    image: 'gitlab/gitlab-ee:latest'
    hostname: 'gitlab.lan'
    environment:
      VIRTUAL_HOST: gitlab.lan,gitlab
    ports:
      - "22:22"
    volumes:
      - '/srv/gitlab/config:/etc/gitlab'
      - '/srv/gitlab/logs:/var/log/gitlab'
      - '/srv/gitlab/data:/var/opt/gitlab'

Still having this issue. Tried just about every combination I can think of in gitlab.rb… I have to be missing something. I’m not supposed to have a separate docker registry running for gitlab to work am I?

I tried enabling the debug port on 5001 as well, but I just get ‘Connection refused’

Also, I’m seeing a bunch of this in the logs:

==> /var/log/gitlab/gitlab-rails/production.log <==
web_1 | Started POST “/v2/home/home-security/blobs/uploads/” for 23.23.x.x at 2021-04-24 06:29:23 +0000
web_1 | Started POST “/v2/home/home-security/blobs/uploads/” for 23.23.x.x at 2021-04-24 06:29:23 +0000
web_1 | Started POST “/v2/home/home-security/blobs/uploads/” for 23.23.x.x at 2021-04-24 06:29:23 +0000
web_1 |
web_1 | ActionController::RoutingError (No route matches [POST] “/v2/home/home-security/blobs/uploads”):
web_1 |
web_1 | lib/gitlab/middleware/basic_health_check.rb:25:in call' web_1 | lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in call’
web_1 | lib/gitlab/middleware/request_context.rb:21:in call' web_1 | config/initializers/fix_local_cache_middleware.rb:11:in call’
web_1 | lib/gitlab/metrics/requests_rack_middleware.rb:76:in call' web_1 | lib/gitlab/middleware/release_env.rb:12:in call’
web_1 |
web_1 | ActionController::RoutingError (No route matches [POST] “/v2/home/home-security/blobs/uploads”):
web_1 |
web_1 | lib/gitlab/middleware/basic_health_check.rb:25:in call' web_1 | lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in call’
web_1 | lib/gitlab/middleware/request_context.rb:21:in call' web_1 | config/initializers/fix_local_cache_middleware.rb:11:in call’
web_1 | lib/gitlab/metrics/requests_rack_middleware.rb:76:in call' web_1 | lib/gitlab/middleware/release_env.rb:12:in call’
web_1 |
web_1 | ActionController::RoutingError (No route matches [POST] “/v2/home/home-security/blobs/uploads”):
web_1 |
web_1 | lib/gitlab/middleware/basic_health_check.rb:25:in call' web_1 | lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in call’
web_1 | lib/gitlab/middleware/request_context.rb:21:in call' web_1 | config/initializers/fix_local_cache_middleware.rb:11:in call’
web_1 | lib/gitlab/metrics/requests_rack_middleware.rb:76:in call' web_1 | lib/gitlab/middleware/release_env.rb:12:in call’
nginx-proxy_1 | nginx.1 | docker.example.io 23.23.x.x - - [24/Apr/2021:06:29:23 +0000] “POST /v2/home/home-security/blobs/uploads/ HTTP/1.1” 404 1576 “-” “docker/20.10.5 go/go1.13.15 git-commit/363e9a8 kernel/5.4.72-microsoft-standard-WSL2 os/linux arch/amd64 UpstreamClient(Docker-Client/20.10.5 \x5C(windows\x5C))”
nginx-proxy_1 | nginx.1 | docker.example.io 23.23.x.x - - [24/Apr/2021:06:29:23 +0000] “POST /v2/home/home-security/blobs/uploads/ HTTP/1.1” 404 1576 “-” “docker/20.10.5 go/go1.13.15 git-commit/363e9a8 kernel/5.4.72-microsoft-standard-WSL2 os/linux arch/amd64 UpstreamClient(Docker-Client/20.10.5 \x5C(windows\x5C))”

So I have no idea how I fixed this. I scrapped my VM I was using to see if somehow I had set some configuration somewhere that was causing this and low and behold…copying the config to a fresh VM and a new docker checkout worked. No more 404s.