Registry.gitlab.com login problem

Error response from daemon: Get https://registry.gitlab.com/v2/: error parsing HTTP 403 response body: unexpected end of JSON input: “”

A 403 error (not 401) indicates the request was a legal request, but the server is refusing to respond to it. Unlike a 401 Unauthorized response, authenticating will make no difference.

I believe the 403 status code is the result of a container registry failed authentication ban.

GitLab.com responds with HTTP status code 403 for 1 hour, if 30 failed authentication requests were received in a 3-minute period from a single IP address.

If you’re still seeing 403 errors, try waiting 1 hour without making any authentication requests to the container registry, and then try again. If the block is removed and the authentication attempt returns a 401 error, check to make sure you’re using the correct username and password/token to authenticate.

4 Likes

Seeing the exact same problem when doing docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" $CI_REGISTRY in my CI pipelie. My Gitlab Docker Registry was working fine as far as I can remember until a while ago (I keep my installation up to date almost weekly). So most likely something changed on Gitlab side as I haven’t touched my Gitlab config. Any clues on how to debug this?

Thanks!

1 Like

I am also having this issue after a recent update to our private gitlab installation.

[GitLab]13.1.3 (8411f47246f)

GitLab Shell13.3.0

GitLab Workhorsev8.35.0

GitLab APIv4

Ruby2.6.6p146

Rails6.0.3.1

PostgreSQL11.7

It seems new tokens being created are not working.

==> /var/log/gitlab/gitlab-rails/production_json.log <==
{"method":"GET","path":"/jwt/auth","format":"html","controller":"JwtController","action":"auth","status":403,"time":"2020-07-10T22:21:13.117Z","params":[{"key":"account","value":"shuat"},{"key":"client_id","value":"docker"},{"key":"offline_token","value":"[FILTERED]"},{"key":"service","value":"container_registry"}],"remote_ip":"*.*.*.*","user_id":null,"username":null,"ua":"docker/19.03.8 go/go1.12.17 git-commit/afacb8b kernel/4.19.76-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.8 \\(darwin\\))","redis_calls":1,"redis_duration_s":0.000289,"redis_read_bytes":1,"redis_write_bytes":57,"redis_cache_calls":1,"redis_cache_duration_s":0.000289,"redis_cache_read_bytes":1,"redis_cache_write_bytes":57,"correlation_id":"NCqv0l37PY9","cpu_s":0.01,"db_duration_s":0.0,"view_duration_s":0.0,"duration_s":0.0028}

==> /var/log/gitlab/gitlab-workhorse/current <==
{"correlation_id":"NCqv0l37PY9","duration_ms":14,"host":"*****","level":"info","method":"GET","msg":"access","proto":"HTTP/1.1","referrer":"","remote_addr":"*.*.*.*:0","remote_ip":"*.*.*.*","status":403,"system":"http","time":"2020-07-10T22:21:13Z","uri":"/jwt/auth?account=shuat\u0026client_id=docker\u0026offline_token=[FILTERED]\u0026service=container_registry","user_agent":"docker/19.03.8 go/go1.12.17 git-commit/afacb8b kernel/4.19.76-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.8 \\(darwin\\))","written_bytes":0}

Above are the logs. How do I troubleshoot from here?

Thanks in advance.

Want to update here for the benefit of others. I tried to login again and it worked.

I am not sure why. If it is because I hit the limit of failed attempts (should not have been more than once) when I used an old token.

I would have expected a message saying that you have been locked out due to too many attempts.

I tried the following commands which used to work to remove blacklisted IPs and it didn’t work.

/opt/gitlab/embedded/bin/redis-cli -s /var/opt/gitlab/redis/redis.socket
del cache:gitlab:rack::attack:allow2ban:ban:<ip>

If anyone cares to shed some light that may be helpful for others.

2 Likes