Docker registry not available

Hello people.

i’ve installed a gitlab 9.2 version with omnibus. And i’ve activated the registry using s3 backend. My problem is that i cant login to docker registry but everything seens to be fine, except this 503 response on the docker login. Any idea?

$ docker login registry.gitlab.mydomain.com
Username: myuser
Password: 
Error response from daemon: Login: {"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
(Code: 503; Headers: map[Server:[nginx] Date:[Thu, 25 May 2017 14:57:48 GMT] Content-Type:[application/json; charset=utf-8] Content-Length:[125]])


$ dpkg -l | grep gitlab
ii  gitlab-ce                           9.2.1-ce.0                         amd64        GitLab Community Edition (including NGINX, Postgres, Redis)
ii  gitlab-ci-multi-runner              9.2.0                              amd64        GitLab Runner

$ gitlab-ctl tail registry
==> /var/log/gitlab/registry/state <==

==> /var/log/gitlab/registry/current <==
2017-05-25_14:59:47.95269 127.0.0.1 - - [25/May/2017:14:59:47 +0000] "GET /v2/ HTTP/1.0" 503 125 "" "docker/17.05.0-ce go/go1.7.5 git-commit/89658be kernel/4.10.15-200.fc25.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/17.05.0-ce \\(linux\\))"
2017-05-25_14:59:48.12088 127.0.0.1 - - [25/May/2017:14:59:48 +0000] "GET /v2/ HTTP/1.0" 503 125 "" "docker/17.05.0-ce go/go1.7.5 git-commit/89658be kernel/4.10.15-200.fc25.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/17.05.0-ce \\(linux\\))"
2017-05-25_14:59:48.29030 127.0.0.1 - - [25/May/2017:14:59:48 +0000] "GET /v1/users/ HTTP/1.0" 503 125 "" "docker/17.05.0-ce go/go1.7.5 git-commit/89658be kernel/4.10.15-200.fc25.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/17.05.0-ce \\(linux\\))"

$ curl -v http://127.0.0.1:5000/v2
* Hostname was NOT found in DNS cache
*   Trying 127.0.0.1...
* Connected to 127.0.0.1 (127.0.0.1) port 5000 (#0)    
> GET /v2 HTTP/1.1
> User-Agent: curl/7.38.0
> Host: 127.0.0.1:5000
> Accept: */*
> 
< HTTP/1.1 503 Service Unavailable
< Content-Type: application/json; charset=utf-8
< Date: Thu, 25 May 2017 15:04:03 GMT
< Content-Length: 125
< 
{"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
* Connection #0 to host 127.0.0.1 left intact

$ gitlab-ctl status registry
run: registry: (pid 10858) 1208s; run: log: (pid 10707) 1241s

I just hit a very similar problem and after some digging it turned out that the container registry health check, which is enabled by default, blows up when the s3 bucket is empty. This appears to be a new issue, since I had not run into it before today.

The solution for me was simply to disable this health check, which doesn’t appear to do anything (someone please correct me if I’m wrong here) anyway. After that the registry worked, I could fill it with container images, and even re-enable the health check if desired. I hope this helps!

Yep, That works! Thanks

Sorry can I know how do you turn out the container registry health check? really thanks!!

This issue still present in gitlab 10.3.1.
for workaround (tested on aws ami)
1 open /opt/gitlab/embedded/cookbooks/registry/templates/default/registry-config.yml.erb
2. set health:storagedriver:enabled to false
3. run gitlab-ctl reconfigure
4. login from docker to your registry
5. push image to some project

Problem is still reproducable at version 10.6.0, but @ghost-x47 workaround still works.

I’ve filed an MR so you can disabled this right from gitlab.rb

registry['storagedriver_health_enabled'] = false

see https://gitlab.com/gitlab-org/omnibus-gitlab/merge_requests/2429

It seems like it should be like this now,

registry['health_storagedriver_enabled'] = false

I had a similar problem, and on verification, I found that the repository was offline.