404 when executing docker push

GitLab CI/CD
1

Hi

I have installed gitlab-ce on my own server and the container-registry was immediately available. When I try to push a docker-image to the container registry I get a:

This text will be blurred1253e8c66eee: Preparing 19b7c4812d37: Preparing 68ecd7dcf510: Preparing c4831d461edd: Preparing a320c397ec46: Preparing f35f5f2080bd: Waiting 58bf93aeb614: Waiting 2f648dff2b08: Waiting bf63529fa390: Waiting 38a1da6431c0: Waiting 97c0c55f9260: Waiting 91dde784976e: Waiting 61bda4f5c10f: Waiting 9a022662d4ac: Waiting 55b0db2111ac: Waiting 60e6c2d9d442: Waiting 2101d3c01933: Waiting 751f5d9ad6db: Waiting 17587239b3df: Waiting 9e63c5bce458: Waiting error parsing HTTP 404 response body: invalid character '<' looking for beginning of value: "<!DOCTYPE html>\n<html>\n<head>\n <meta content=\"width=device-width, initial-scale=1, maximum-scale=1\" name=\"viewport\">\n <title>The page you're looking for could not be found (404)</title>\n <style>\n body {\n color: #666;\n text-align: center;\n font-family: \"Helvetica Neue\", Helvetica, Arial, sans-serif;\n margin: auto;\n font-size: 14px;\n }\n\n h1 {\n font-size: 56px;\n line-height: 100px;\n font-weight: 400;\n color: #456;\n }\n\n h2 {\n font-size: 24px;\n color: #666;\n line-height: 1.5em;\n }\n\n h3 {\n color: #456;\n font-size: 20px;\n font-weight: 400;\n line-height: 28px;\n }\n\n hr {\n max-width: 800px;\n margin: 18px auto;\n border: 0;\n border-top: 1px solid #EEE;\n border-bottom: 1px solid white;\n }\n\n img {\n max-width: 40vw;\n display: block;\n margin: 40px auto;\n }\n\n a {\n line-height: 100px;\n font-weight: 400;\n color: #4A8BEE;\n font-size: 18px;\n text-decoration: none;\n }\n\n .container {\n margin: auto 20px;\n }\n\n .go-back {\n display: none;\n }\n\n </style>\n</head>\n\n<body>\n <a href=\"/\">\n <img src=\"data:image/svg+xml;base64,PHN2ZyB3aWR0aD0iMjEwIiBoZWlnaHQ9IjIxMCIgdmlld0JveD0iMCAwIDIxMCAyMTAiIHhtbG5zPSJodHRwOi8vd3d3LnczLm9yZy8yMDAwL3N2ZyI+CiAgPHBhdGggZD0iTTEwNS4wNjE0IDIwMy42NTVsMzguNjQtMTE4LjkyMWgtNzcuMjhsMzguNjQgMTE4LjkyMXoiIGZpbGw9IiNlMjQzMjkiLz4KICA8cGF0aCBkPSJNMTA1LjA2MTQgMjAzLjY1NDhsLTM4LjY0LTExOC45MjFoLTU0LjE1M2w5Mi43OTMgMTE4LjkyMXoiIGZpbGw9IiNmYzZkMjYiLz4KICA8cGF0aCBkPSJNMTIuMjY4NSA4NC43MzQxbC0xMS43NDIgMzYuMTM5Yy0xLjA3MSAzLjI5Ni4xMDIgNi45MDcgMi45MDYgOC45NDRsMTAxLjYyOSA3My44MzgtOTIuNzkzLTExOC45MjF6IiBmaWxsPSIjZmNhMzI2Ii8+CiAgPHBhdGggZD0iTTEyLjI2ODUgODQuNzM0Mmg1NC4xNTNsLTIzLjI3My03MS42MjVjLTEuMTk3LTMuNjg2LTYuNDExLTMuNjg1LTcuNjA4IDBsLTIzLjI3MiA3MS42MjV6IiBmaWxsPSIjZTI0MzI5Ii8+CiAgPHBhdGggZD0iTTEwNS4wNjE0IDIwMy42NTQ4bDM4LjY0LTExOC45MjFoNTQuMTUzbC05Mi43OTMgMTE4LjkyMXoiIGZpbGw9IiNmYzZkMjYiLz4KICA8cGF0aCBkPSJNMTk3Ljg1NDQgODQuNzM0MWwxMS43NDIgMzYuMTM5YzEuMDcxIDMuMjk2LS4xMDIgNi45MDctMi45MDYgOC45NDRsLTEwMS42MjkgNzMuODM4IDkyLjc5My0xMTguOTIxeiIgZmlsbD0iI2ZjYTMyNiIvPgogIDxwYXRoIGQ9Ik0xOTcuODU0NCA4NC43MzQyaC01NC4xNTNsMjMuMjczLTcxLjYyNWMxLjE5Ny0zLjY4NiA2LjQxMS0zLjY4NSA3LjYwOCAwbDIzLjI3MiA3MS42MjV6IiBmaWxsPSIjZTI0MzI5Ii8+Cjwvc3ZnPgo=\"\n alt=\"GitLab Logo\" />\n </a>\n <h1>\n 404\n </h1>\n <div class=\"container\">\n <h3>The page could not be found or you don't have permission to view it.</h3>\n <hr />\n <p>The resource that you are attempting to access does not exist or you don't have the necessary permissions to view it.</p>\n <p>Make sure the address is correct and that the page hasn't moved.</p>\n <p>Please contact your GitLab administrator if you think this is a mistake.</p>\n <a href=\"javascript:history.back()\" class=\"js-go-back go-back\">Go back</a>\n </div>\n <script>\n (function () {\n var goBack = document.querySelector('.js-go-back');\n\n if (history.length > 1) {\n goBack.style.display = 'inline';\n }\n })();\n </script>\n</body>\n</html>\n"

what is that? I mean I can login successfully, so why the 404?

in my gitlab.rb I have

external_url 'https://git.xxxxxxxx.com'
nginx['enable'] = true
nginx['client_max_body_size'] = '250m'
nginx['redirect_http_to_https'] = true

nginx['ssl_certificate'] = "/etc/gitlab/trusted-certs/xxxxxxxx.com.crt"
nginx['ssl_certificate_key'] = "/etc/gitlab/trusted-certs/xxxxxxxx.com.key"
nginx['ssl_protocols'] = "TLSv1.1 TLSv1.2"

in the config.yml i have:

version: 0.1
log:
  level: info
  formatter: text
  fields:
    service: registry
    environment: production
storage: {"filesystem":{"rootdirectory":"/var/opt/gitlab/gitlab-rails/shared/registry"},"cache": {"blobdescriptor":"inmemory"},"delete":{"enabled":true}}
http:
  addr: localhost:5000
  secret: "xxxxxxxxxxxx"
  headers:
    X-Content-Type-Options: [nosniff]
health:
  storagedriver:
    enabled: true
    interval: 10s
    threshold: 3
auth:
  token:
    realm: https://git.latronic.com/jwt/auth
    service: container_registry
    issuer: omnibus-gitlab-issuer
    rootcertbundle: /var/opt/gitlab/registry/gitlab-registry.crt
    autoredirect: false
validation:
  disabled: true
2

Hi Y’all – I also am having this issue: docker login succeeds, the push fails with this 404 error (both from the command line and as part of a CI process). I’ve experienced this behavior with both v12.10 and v13.2; with my user/pass and also user/token (as well as nonce CI credentials). I’m the owner of the repository (“docker/testbuild” so no special characters or anything there) and an admin of the GitLab CE installation. My configuration with regard to the container registry is the installation default. Any advice would be appreciated! Thanks.

3

Following up my own post:

It turns out I didn’t read the documentation throughly enough (or my reading comprehension was sub-optimal). Uncommenting these parameters and re-configuring GitLab seems to have solved my issue:

registry_external_url 'https://gitlab.example.com:5050'

### Settings used by GitLab application
gitlab_rails['registry_enabled'] = true
gitlab_rails['registry_host'] = "gitlab.example.com"
gitlab_rails['registry_port'] = "5050"

This all makes sense really. There’s no reason a docker push to an HTTPS port should do anything useful.

4

Hi, I am having the same problem is there something specific you are doing with your reverse proxy i cant find the problem?

5

I struggled with this recently and was able to solve it. Context:

  • Docker CE installation
  • nginx system proxy (= nginx installed on the docker host)

Both registry and gitlab work with SSL with the configs below for this context.

The docker-compose.yml:

version: '3'

services:
  web:
    image: gitlab/gitlab-ce:latest
    restart: always
    environment:
      GITLAB_OMNIBUS_CONFIG: |
        sidekiq['max_concurrency'] = 10
        prometheus_monitoring['enable'] = false
        gitlab_rails['registry_enabled'] = true
        gitlab_rails['registry_host'] = "http://registry.local.mydomain.tld"
        external_url 'https://gitlab.local.mydomain.tld'
        nginx['listen_port'] = 80
        nginx['listen_https'] = false
        letsencrypt['enable'] = false
        gitlab_rails['gitlab_shell_ssh_port'] = 9999
        registry_external_url 'http://registry.local.mydomain.tld/'
        registry_nginx['listen_https'] = false
        registry_nginx['proxy_set_headers'] = {
            "X-Forwarded-Proto" => "https",
            "X-Forwarded-Ssl" => "on"
        }
    ports:
        - '31080:80'
        - '31088:8080'
        - '9999:22'
    volumes:
        - '/srv/gitlab/config:/etc/gitlab'
        - '/srv/gitlab/logs:/var/log/gitlab'
        - '/srv/gitlab/data:/var/opt/gitlab'

  watchtower:
    image: containrrr/watchtower
    restart: always
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    command: --interval 86400

The nginx gitlab.conf:

server {
    listen 443 ssl http2;
    listen [::]:443 ssl http2;
    server_name gitlab.local.mydomain.tld;

    # SSL
    ssl_certificate     /etc/nginx/ssl/wildcard.local.mydomain.tld.fullchain;
    ssl_certificate_key /etc/nginx/ssl/wildcard.local.mydomain.tld.key;
    # ssl_trusted_certificate /etc/letsencrypt/live/exam

    # logging
    access_log /var/log/nginx/gitlab.local.mydomain.tld.access.log;
    error_log /var/log/nginx/gitlab.local.mydomain.tld.error.log warn;

    # reverse proxy
       location / {
       proxy_read_timeout      300;
       proxy_connect_timeout   300;
       proxy_redirect          off;

       proxy_set_header        Host                $http_host;
       proxy_set_header        X-Real-IP           $remote_addr;
       proxy_set_header        X-Forwarded-For     $proxy_add_x_forwarded_for;
       proxy_set_header        X-Forwarded-Proto   https;
       proxy_set_header        X-Frame-Options     SAMEORIGIN;
       proxy_pass http://127.0.0.1:31080;
  }
}

server {
  listen 80;
  listen [::]:80;
  listen 443 ssl;
  
  server_name registry.local.mydomain.tld;

  ssl_certificate     /etc/nginx/ssl/wildcard.local.mydomain.tld.fullchain;
  ssl_certificate_key /etc/nginx/ssl/wildcard.local.mydomain.tld.key;

  location / {
    proxy_read_timeout      300;
    proxy_connect_timeout   300;
    proxy_redirect          off;

    proxy_set_header        Host                $http_host;
    proxy_set_header        X-Real-IP           $remote_addr;
    proxy_set_header        X-Forwarded-For     $proxy_add_x_forwarded_for;
    proxy_set_header        X-Forwarded-Proto   https;
    proxy_set_header        X-Frame-Options     SAMEORIGIN;
    proxy_pass http://127.0.0.1:31080;
  }

  if ($https = '') { return 301 https://$host$request_uri; }  # if not connected to HTTPS, perma-redirect to HTTPS
}

As you can see:

  • I use a dedicated subdomain for the registry (the subdomain is not a child of gitlab - this is possible, but not a must)
  • My SSL certs are offloaded at nginx, meaning that all traffic to gitlab and the registry are forwarded using http (from the docker host to the docker container)
  • nginx forwards traffic for both gitlab and the registry to the same Docker port 80 (using local port 31080). The gitlab container knows from the header to assign traffic to gitlab or the registry

Specifically, what caused the 404 error for me on docker push was that I had nginx set up correctly, but:

  • did not enable the registry
  • had https:// set for traffic to the registry
gitlab_rails['registry_enabled'] = true
gitlab_rails['registry_host'] = "http://registry.local.mydomain.tld"
registry_external_url 'http://registry.local.mydomain.tld/'
registry_nginx['listen_https'] = false

Conclusion: If you’re doing SSL offloading at the reverse-proxy, make sure you disable any SSL at the Gitlab Docker container.

Two notes:

  • This is an ‘internal’ SSL setup, meaning that my Gitlab can only be reached from the LAN side. however, for the setup, this really does not matter. The configs above would work the same in a public setting.
  • Since I wanted to preserve the SSH port 22 to connect to my LXC, my repositories are all made available on a different port, meaning that I clone my repositories with this specific port, e.g. ssh://git@gitlab.local.mydomain.tld:9999/configs/nginx-docker-gl.git