Problem to solve
Trying to docker-login to the GitLab Registry (self-hosted) returns a 400.
$ docker login registry.example.org
Username: my-user
Password:
Error response from daemon: login attempt to https://registry.example.org/v2/ failed with status: 400 Bad Request
I would expect to be able to login with my credentials, or at least not have a 400.
I use MFA for my GitLab account, so I use a Personal Token for the authentication.
If I redirect “registry.example.org” to my “gitlab” container on the 443 port, it works properly, but then, it is the “push” that fails.
Steps to reproduce
My docker-compose.yml have this block:
gitlab:
image: gitlab/gitlab-ce:latest
container_name: GitLab
hostname: gitlab.${HOSTNAME}
restart: unless-stopped
volumes:
- /srv/gitlab:/var/opt/gitlab:z
- ./gitlab-conf:/etc/gitlab:z
environment:
GITLAB_OMNIBUS_CONFIG: |
external_url 'https://gitlab.example.org';
mattermost_external_url 'https://mm.example.org';
registry_external_url 'https://registry.example.org';
letsencrypt['enable'] = false;
gitlab_rails['smtp_enable'] = true;
gitlab_rails['smtp_address'] = '${SMTP_ADDRESS}';
gitlab_rails['smtp_port'] = ${SMTP_PORT};
gitlab_rails['smtp_user_name'] = '${SMTP_USERNAME}';
gitlab_rails['smtp_password'] = '${SMTP_PASSWORD}';
gitlab_rails['smtp_domain'] = '${SMTP_DOMAIN}';
gitlab_rails['smtp_authentication'] = 'login';
gitlab_rails['smtp_tls'] = true;
gitlab_rails['time_zone'] = 'America/Montreal';
gitlab_rails['gitlab_email_enabled'] = true;
gitlab_rails['gitlab_email_from'] = '${GITLAB_EMAIL_FROM}';
gitlab_rails['gitlab_email_display_name'] = '${GITLAB_EMAIL_DISPLAY}';
gitlab_rails['gitlab_email_reply_to'] = '${GITLAB_EMAIL_REPLYTO}';
gitlab_rails['gitlab_email_subject_suffix'] = '[GitLab] ';
gitlab_rails['gitlab_default_theme'] = 3;
gitlab_rails['registry_enabled'] = true;
registry['enable'] = true;
registry['token_realm'] = "https://gitlab.example.org";
registry['env'] = { "REGISTRY_HTTP_RELATIVEURLS" => true };
registry_nginx['enable'] = true;
registry_nginx['listen_port'] = 5050;
postgresql['shared_buffers'] = "512MB";
postgresql['work_mem'] = "64MB";
mattermost_nginx['gitlab_enable'] = true;
mattermost_nginx['gitlab_id'] = '${MATTERMOST_ID}';
mattermost_nginx['gitlab_secret'] = '${MATTERMOST_SECRET}';
mattermost_nginx['gitlab_scope'] = '';
mattermost_nginx['gitlab_auth_endpoint'] = 'https://gitlab.example.org/oauth/authorize';
mattermost_nginx['gitlab_token_endpoint'] = 'https://gitlab.example.org/oauth/token';
mattermost_nginx['gitlab_user_api_endpoint'] = 'https://gitlab.example.org/api/v4/user';
mattermost['service_use_ssl'] = true;
mattermost['env'] = {
'MM_SERVICESETTINGS_SITEURL' => 'https://mm.example.org/',
'MM_SERVICESETTINGS_ENABLELINKPREVIEWS' => 'true',
"MM_SERVICESETTINGS_ALLOWEDUNTRUSTEDINTERNALCONNECTIONS": " gitlab.example.org",
'MM_EMAILSETTINGS_ENABLESIGNINWITHEMAIL' => 'false',
'MM_EMAILSETTINGS_ENABLESMTPAUTH' => 'true',
'MM_EMAILSETTINGS_SMTPSERVER' => '${SMTP_ADDRESS}',
'MM_EMAILSETTINGS_SMTPPORT' => ${SMTP_PORT},
'MM_EMAILSETTINGS_SMTPUSERNAME' => '${SMTP_USERNAME}',
'MM_EMAILSETTINGS_SMTPPASSWORD' => '${SMTP_PASSWORD}',
'MM_EMAILSETTINGS_SMTPDOMAIN' => '${SMTP_DOMAIN}',
'MM_EMAILSETTINGS_CONNECTIONSECURITY' => 'TLS',
'MM_EMAILSETTINGS_SENDEMAILNOTIFICATIONS' => 'true',
'MM_EMAILSETTINGS_ENABLEEMAILBATCHING' => 'true',
'MM_EMAILSETTINGS_SENDPUSHNOTIFICATIONS' => 'true',
'MM_EMAILSETTINGS_PUSHNOTIFICATIONSERVER' => 'https://push-test.mattermost.com',
'MM_EMAILSETTINGS_FEEDBACKNAME' => 'MatterMost Notification (mm.example.org)',
'MM_EMAILSETTINGS_FEEDBACKORGANIZATION' => '© ACME Corp.',
'MM_EMAILSETTINGS_FEEDBACKEMAIL' => '${GITLAB_EMAIL_REPLYTO}',
'MM_THEMESETTINGS_DEFAULTTHEME' => 'Organization',
'MM_TEAMSETTINGS_ENABLETEAMCREATION' => 'false',
'MM_SUPPORTSETTINGS_SUPPORTEMAIL' => '${GITLAB_EMAIL_REPLYTO}',
'MM_GITLABSETTINGS_ENABLE' => 'true',
'MM_GITLABSETTINGS_SECRET' => '${MATTERMOST_SECRET}',
'MM_GITLABSETTINGS_ID' => '${MATTERMOST_ID}',
'MM_GITLABSETTINGS_SCOPE' => '',
'MM_GITLABSETTINGS_AUTHENDPOINT' => 'https://gitlab.example.org/oauth/authorize',
'MM_GITLABSETTINGS_TOKENENDPOINT' => 'https://gitlab.example.org/oauth/token',
'MM_GITLABSETTINGS_USERAPIENDPOINT' => 'https://gitlab.example.org/api/v4/user'
};
I have a nGinX proxy in front of GitLab/MatterMost/Registry.
The GitLab server entry has the following location and works properly:
location / {
access_log off;
proxy_pass https://gitlab;
proxy_redirect off;
proxy_buffering off;
proxy_http_version 1.1;
proxy_pass_header Set-Cookie;
proxy_set_header Connection "Keep-Alive";
proxy_set_header Proxy-Connection "Keep-Alive";
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-Host $http_host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
break;
}
The MatterMost proxy also works properly.
The Registry proxy is the following, for the registry.example.org domain:
location / {
access_log off;
proxy_pass https://gitlab:5050;
proxy_redirect off;
# proxy_set_header Upgrade $http_upgrade;
# proxy_set_header Connection "upgrade";
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Forwarded-Protocol $scheme;
proxy_set_header X-Forwarded-Schema $scheme;
proxy_set_header X-Forwarded-Host $http_host;
proxy_set_header X-Url-Scheme $scheme;
# proxy_buffers 256 16k;
# proxy_buffer_size 16k;
# client_body_timeout 60;
# client_max_body_size 50M;
# send_timeout 300;
# lingering_timeout 5;
# proxy_connect_timeout 90;
# proxy_send_timeout 300;
# proxy_read_timeout 90s;
}
I’ve read a lot of posts and tried a lot of thing, this is the state of it right now.
$ docker login registry.example.org
Username: my-user
Password:
Error response from daemon: login attempt to https://registry.example.org/v2/ failed with status: 400 Bad Request
I’ve also noted that going to “https://registry.example.org/v2/” from my browser, in the “response headers”, I see www-authenticate: Bearer realm="https://gitlab.example.org/jwt/auth",service="container_registry"
But if I navigate to https://gitlab.example.org/jwt/, I get a 404.
If I change proxy_pass https://gitlab:5050; to proxy_pass https://gitlab;, it will authenticate correctly.
But uploading the docker image will fail with a “unauthenticated” error, looking at the logs, I can see 401 for the “blob/uploads” calls.
$ docker push registry.example.org/group/project
Using default tag: latest
The push refers to repository [registry.example.org/group/project]
6c128582b98f: Preparing
528112c685eb: Preparing
1c88035fc2d3: Preparing
376d46f4ac8b: Preparing
9bc977164420: Preparing
c62134047f95: Waiting
5f70bf18a086: Waiting
c8bbb5abd45e: Waiting
unauthorized: authentication required
Configuration
# gitlab-ctl status
run: alertmanager: (pid 1093) 1091s; run: log: (pid 938) 1145s
run: gitaly: (pid 304) 1327s; run: log: (pid 324) 1324s
run: gitlab-exporter: (pid 1074) 1093s; run: log: (pid 826) 1166s
run: gitlab-kas: (pid 468) 1310s; run: log: (pid 481) 1307s
run: gitlab-workhorse: (pid 1037) 1095s; run: log: (pid 536) 1288s
run: logrotate: (pid 267) 1340s; run: log: (pid 277) 1337s
run: mattermost: (pid 1055) 1094s; run: log: (pid 777) 1172s
run: nginx: (pid 550) 1286s; run: log: (pid 562) 1283s
run: postgres-exporter: (pid 1101) 1091s; run: log: (pid 994) 1140s
run: postgresql: (pid 331) 1316s; run: log: (pid 434) 1313s
run: prometheus: (pid 1084) 1092s; run: log: (pid 917) 1152s
run: puma: (pid 484) 1304s; run: log: (pid 493) 1301s
run: redis: (pid 280) 1334s; run: log: (pid 295) 1331s
run: redis-exporter: (pid 1076) 1093s; run: log: (pid 895) 1159s
run: registry: (pid 1045) 1094s; run: log: (pid 608) 1277s
run: sidekiq: (pid 502) 1298s; run: log: (pid 512) 1295s
run: sshd: (pid 34) 1355s; run: log: (pid 33) 1355s
Versions
Please select whether options apply, and add the version information.
- Self-managed
- GitLab.com SaaS
Versions
- GitLab Community Edition v17.3.1