Hello, every 01!
I can’t get docker login to work correctly with gitlab and a registry using docker-compose.
When I log in with a wrong password I see
% docker login -u georg -p wrong registry.mydomain.org
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
Error response from daemon: Get "https://registry.mydomain.org/v2/": unauthorized: HTTP Basic: Access denied
When I log in with my gitlab password I see
% docker login -u georg -p right registry.mydomain.org
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
Error response from daemon: login attempt to https://registry.mydomain.org/v2/ failed with status: 401 Unauthorized
The same happens when I log in at the registry’s backend address (i have added "insecure-registries": ["gitlab04.mydomain.org:5000"]
to the docker configuration on the machine I’m using to try)
Wrong:
% docker login -u georg -p wrong http://gitlab04.mydomain.org:5000
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
Error response from daemon: Get "http://gitlab04.mydomain.org:5000/v2/": unauthorized: HTTP Basic: Access denied
Right:
% docker login -u georg -p right http://gitlab04.mydomain.org:5000
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
Error response from daemon: login attempt to http://gitlab04.mydomain.org:5000/v2/ failed with status: 401 Unauthorized
Here’s the setup I’m using:
git.mydomain.org is accessible via https
registry.mydomain.org is accessible via https
nginx handles both URLs as a reverse proxy
gitlab04.mydomain.org is running gitlab, the registry and a runner
git.mydomain.org and registry.mydomain.org are being handled by nginx, here’s its configuration:
/etc/nginx/sites-available/git.mydomain.org.conf
upstream gitlab {
server gitlab04.mydomain.org:80 fail_timeout=0;
}
server {
if ($host = git.mydomain.org) {
return 301 https://$host$request_uri;
} # managed by Certbot
listen 80;
server_name git.mydomain.org;
return 301 https://$server_name$request_uri;
}
# let gitlab deal with the redirection
server {
listen 443 ssl;
server_name git.mydomain.org;
root /dev/null;
location / {
gzip off;
proxy_read_timeout 300;
proxy_connect_timeout 300;
proxy_redirect off;
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto https;
proxy_set_header X-Frame-Options SAMEORIGIN;
proxy_pass http://gitlab;
}
ssl_certificate /etc/letsencrypt/live/git.mydomain.org/fullchain.pem; # managed by Certbot
ssl_certificate_key /etc/letsencrypt/live/git.mydomain.org/privkey.pem; # managed by Certbot
}
/etc/nginx/sites-available/registry.mydomain.org.conf
server {
if ($host = registry.mydomain.org) {
return 301 https://$host$request_uri;
} # managed by Certbot
listen 80;
server_name registry.mydomain.org;
return 301 https://$server_name$request_uri;
}
server {
listen 443 ssl;
server_name registry.mydomain.org;
ssl_ciphers 'ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA256:ECDHE-RSA-AES256-SHA:ECDHE-RSA-AES128-SHA:ECDHE-RSA-DES-CBC3-SHA:AES256-GCM-SHA384:AES128-GCM-SHA256:AES256-SHA256:AES128-SHA256:AES256-SHA:AES128-SHA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!MD5:!PSK:!RC4';
ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
ssl_prefer_server_ciphers on;
ssl_session_cache builtin:1000 shared:SSL:10m;
ssl_session_timeout 5m;
client_max_body_size 0;
chunked_transfer_encoding on;
location / {
# Do not allow connections from docker 1.5 and earlier
# docker pre-1.6.0 did not properly set the user agent on ping, catch "Go *" user agents
if ($http_user_agent ~ "^(docker\/1\.(3|4|5(?!\.[0-9]-dev))|Go ).*$" ) {
return 404;
}
proxy_pass http://gitlab04.mydomain.org:5000;
proxy_set_header Host $http_host; # required for docker client's sake
proxy_set_header X-Real-IP $remote_addr; # pass on real client's IP
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto https;
proxy_read_timeout 900;
}
ssl_certificate /etc/letsencrypt/live/registry.mydomain.org/fullchain.pem; # managed by Certbot
ssl_certificate_key /etc/letsencrypt/live/registry.mydomain.org/privkey.pem; # managed by Certbot
}
I copied the currentletsencrypt registry certificate and key over to the backend and “fixed” their permissions. There was an “Internal Server Error” because the key file couldn’t be read by gitlab.
This is the docker-compose.yaml
version: "3.9"
services:
gitlab:
image: 'ulm0/gitlab:13.2.6'
# image: 'ulm0/gitlab:13.10.0'
container_name: 'gitlab'
hostname: 'git.mydomain.org'
environment:
GITLAB_OMNIBUS_CONFIG: |
external_url 'https://git.mydomain.org'
registry_external_url 'https://registry.mydomain.org'
nginx['listen_port'] = 80
nginx['listen_https'] = false
letsencrypt['enable'] = false
registry['enable'] = false
gitlab_rails['registry_enabled'] = true
gitlab_rails['registry_api_url'] = "https://registry.mydomain.org"
gitlab_rails['registry_key_path'] = "/certs/registry.key"
gitlab_rails['registry_host'] = "registry.mydomain.org"
gitlab_rails['registry_port'] = "443"
ports:
- '80:80'
volumes:
- '/srv/gitlab/config:/etc/gitlab'
- '/srv/gitlab/logs:/var/log/gitlab'
- '/srv/gitlab/data:/var/opt/gitlab'
- '/srv/certs/:/certs'
restart: 'unless-stopped'
registry:
image: registry:latest
container_name: 'registry'
volumes:
- '/srv/registry/data:/registry'
- '/srv/certs:/certs'
environment:
- REGISTRY_LOG_LEVEL=info
- REGISTRY_STORAGE_FILESYSTEM_ROOTDIRECTORY=/registry
- REGISTRY_AUTH_TOKEN_REALM=https://git.mydomain.org/jwt/auth
- REGISTRY_AUTH_TOKEN_SERVICE=container_registry
- REGISTRY_AUTH_TOKEN_ISSUER=gitlab-issuer
- REGISTRY_AUTH_TOKEN_ROOTCERTBUNDLE=/certs/registry.crt
- REGISTRY_STORAGE_DELETE_ENABLED=true
ports:
- '5000:5000'
restart: 'unless-stopped'
runner:
image: gitlab/gitlab-runner:alpine
container_name: 'runner'
volumes:
- '/var/run/docker.sock:/var/run/docker.sock'
- '/srv/runner/cache/:/cache'
- '/srv/runner/etc/:/etc/gitlab-runner'
restart: 'unless-stopped'
I realized that the file gitlab-secrets.json
had invalid entries for “registry”, because I restored a backup of my former gitlab instance. I removed the entry and reconfigured. Unfortunately that didn’t resolve my problem. I see the same login error happen when the runner is picking up a job and running through a default template for CI/CD that uses the registry.
$ docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" $CI_REGISTRY
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
Error response from daemon: login attempt to https://registry.mydomain.org:443/v2/ failed with status: 401 Unauthorized
ERROR: Job failed: exit code 1
I have no idea what could be wrong. I’m getting the very same error when I use an access token as password. I’m pretty much stuck now and any help would be very appreciated.