Docker login 401 Unauthorized, external registry

Hello, every 01!

I can’t get docker login to work correctly with gitlab and a registry using docker-compose.

When I log in with a wrong password I see

% docker login -u georg -p wrong registry.mydomain.org                  
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
Error response from daemon: Get "https://registry.mydomain.org/v2/": unauthorized: HTTP Basic: Access denied

When I log in with my gitlab password I see

% docker login -u georg -p right registry.mydomain.org
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
Error response from daemon: login attempt to https://registry.mydomain.org/v2/ failed with status: 401 Unauthorized

The same happens when I log in at the registry’s backend address (i have added "insecure-registries": ["gitlab04.mydomain.org:5000"] to the docker configuration on the machine I’m using to try)

Wrong:

% docker login -u georg -p wrong http://gitlab04.mydomain.org:5000
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
Error response from daemon: Get "http://gitlab04.mydomain.org:5000/v2/": unauthorized: HTTP Basic: Access denied

Right:

% docker login -u georg -p right http://gitlab04.mydomain.org:5000
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
Error response from daemon: login attempt to http://gitlab04.mydomain.org:5000/v2/ failed with status: 401 Unauthorized

Here’s the setup I’m using:

git.mydomain.org is accessible via https
registry.mydomain.org is accessible via https
nginx handles both URLs as a reverse proxy
gitlab04.mydomain.org is running gitlab, the registry and a runner

git.mydomain.org and registry.mydomain.org are being handled by nginx, here’s its configuration:

/etc/nginx/sites-available/git.mydomain.org.conf

upstream gitlab {
    server                    gitlab04.mydomain.org:80 fail_timeout=0;
}

server {
    if ($host = git.mydomain.org) {
        return 301 https://$host$request_uri;
    } # managed by Certbot

    listen          80;
    server_name     git.mydomain.org;
    return          301 https://$server_name$request_uri;
}

# let gitlab deal with the redirection
server {
    listen                      443 ssl;
    server_name                 git.mydomain.org;
    root                        /dev/null;

    location / {
        gzip                    off;
        proxy_read_timeout      300;
        proxy_connect_timeout   300;
        proxy_redirect          off;
        proxy_set_header        Host                $http_host;
        proxy_set_header        X-Real-IP           $remote_addr;
        proxy_set_header        X-Forwarded-For     $proxy_add_x_forwarded_for;
        proxy_set_header        X-Forwarded-Proto   https;
        proxy_set_header        X-Frame-Options     SAMEORIGIN;
        proxy_pass      http://gitlab;
    }

    ssl_certificate /etc/letsencrypt/live/git.mydomain.org/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/git.mydomain.org/privkey.pem; # managed by Certbot
}

/etc/nginx/sites-available/registry.mydomain.org.conf

server {
    if ($host = registry.mydomain.org) {
        return 301 https://$host$request_uri;
    } # managed by Certbot

    listen          80;
    server_name     registry.mydomain.org;
    return          301 https://$server_name$request_uri;
}

server {
    listen 443 ssl;
    server_name registry.mydomain.org;

    ssl_ciphers 'ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA256:ECDHE-RSA-AES256-SHA:ECDHE-RSA-AES128-SHA:ECDHE-RSA-DES-CBC3-SHA:AES256-GCM-SHA384:AES128-GCM-SHA256:AES256-SHA256:AES128-SHA256:AES256-SHA:AES128-SHA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!MD5:!PSK:!RC4';
    ssl_protocols  TLSv1 TLSv1.1 TLSv1.2;
    ssl_prefer_server_ciphers on;
    ssl_session_cache  builtin:1000  shared:SSL:10m;
    ssl_session_timeout  5m;

    client_max_body_size 0;
    chunked_transfer_encoding on;

    location / {
      # Do not allow connections from docker 1.5 and earlier
      # docker pre-1.6.0 did not properly set the user agent on ping, catch "Go *" user agents
      if ($http_user_agent ~ "^(docker\/1\.(3|4|5(?!\.[0-9]-dev))|Go ).*$" ) {
        return 404;
      }

      proxy_pass                          http://gitlab04.mydomain.org:5000;
      proxy_set_header  Host              $http_host;   # required for docker client's sake
      proxy_set_header  X-Real-IP         $remote_addr; # pass on real client's IP
      proxy_set_header  X-Forwarded-For   $proxy_add_x_forwarded_for;
      proxy_set_header  X-Forwarded-Proto https;
      proxy_read_timeout                  900;
    }

    ssl_certificate /etc/letsencrypt/live/registry.mydomain.org/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/registry.mydomain.org/privkey.pem; # managed by Certbot
}

I copied the currentletsencrypt registry certificate and key over to the backend and “fixed” their permissions. There was an “Internal Server Error” because the key file couldn’t be read by gitlab.

This is the docker-compose.yaml

version: "3.9"

services:

  gitlab:
    image: 'ulm0/gitlab:13.2.6'
    # image: 'ulm0/gitlab:13.10.0'
    container_name: 'gitlab'
    hostname: 'git.mydomain.org'
    environment:
      GITLAB_OMNIBUS_CONFIG: |
          external_url 'https://git.mydomain.org'
          registry_external_url 'https://registry.mydomain.org'
          nginx['listen_port'] = 80
          nginx['listen_https'] = false
          letsencrypt['enable'] = false
          registry['enable'] = false
          gitlab_rails['registry_enabled'] = true
          gitlab_rails['registry_api_url'] = "https://registry.mydomain.org"
          gitlab_rails['registry_key_path'] = "/certs/registry.key"
          gitlab_rails['registry_host'] = "registry.mydomain.org"
          gitlab_rails['registry_port'] = "443"
    ports:
      - '80:80'
    volumes:
      - '/srv/gitlab/config:/etc/gitlab'
      - '/srv/gitlab/logs:/var/log/gitlab'
      - '/srv/gitlab/data:/var/opt/gitlab'
      - '/srv/certs/:/certs'
    restart: 'unless-stopped'

  registry:
    image: registry:latest
    container_name: 'registry'
    volumes:
      - '/srv/registry/data:/registry'
      - '/srv/certs:/certs'
    environment:
      - REGISTRY_LOG_LEVEL=info
      - REGISTRY_STORAGE_FILESYSTEM_ROOTDIRECTORY=/registry
      - REGISTRY_AUTH_TOKEN_REALM=https://git.mydomain.org/jwt/auth
      - REGISTRY_AUTH_TOKEN_SERVICE=container_registry
      - REGISTRY_AUTH_TOKEN_ISSUER=gitlab-issuer
      - REGISTRY_AUTH_TOKEN_ROOTCERTBUNDLE=/certs/registry.crt
      - REGISTRY_STORAGE_DELETE_ENABLED=true
    ports:
      - '5000:5000'
    restart: 'unless-stopped'

  runner:
    image: gitlab/gitlab-runner:alpine
    container_name: 'runner'
    volumes:
      - '/var/run/docker.sock:/var/run/docker.sock'
      - '/srv/runner/cache/:/cache'
      - '/srv/runner/etc/:/etc/gitlab-runner'
    restart: 'unless-stopped'

I realized that the file gitlab-secrets.json had invalid entries for “registry”, because I restored a backup of my former gitlab instance. I removed the entry and reconfigured. Unfortunately that didn’t resolve my problem. I see the same login error happen when the runner is picking up a job and running through a default template for CI/CD that uses the registry.

$ docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" $CI_REGISTRY
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
Error response from daemon: login attempt to https://registry.mydomain.org:443/v2/ failed with status: 401 Unauthorized
ERROR: Job failed: exit code 1

I have no idea what could be wrong. I’m getting the very same error when I use an access token as password. I’m pretty much stuck now and any help would be very appreciated.

For future reference: After some hours of digging into this i found that gitlab will overwrite the registry.key file on startup with a key file stored in gitlab-secrets.json - I migrated this instance of gitlab that’s why there were entries.

So: I created a new cert/key pair with

openssl req -x509 -newkey rsa:4096 -keyout /srv/certs/token.key -out /srv/certs/token.crt -days 36500 -nodes

I used FQDN: registry.mydomain.org
Then i transfered the key and crt contents to gitlab-secrets.json and I got auth working! I’m now also using the REGISTRY_HTTP_SECRET i found in gitlab-secrets.json, now it works.