Container registry image repository sporadically shows invalid tag: missing manifest digest

How to Use GitLab
1

My organisation has been using a self-managed community edition of GitLab for a while now and we decided to switch from the sameersbn/gitlab:14.8.2 docker image to Kubernetes, by using the gitlab Helm-Chart (Chart version 5.8.2, which is GitLab v14.8.2). Initial installation and the backup and restore of our former data was tricky, but turned out fine.

We have since tested this Kubernetes installation and have found a problem with the container registry. It is randomly showing the “Invalid tag: missing manifest digest” error, as you can see below:

gitlab
gitlab1919×960 79.6 KB

This error occurs for our old images from the backup & restore procedure, as well as images we pushed recently.

While the error is shown in the GUI, any attempt at pulling the affected images results in Error response from daemon: received unexpected HTTP status: 500 Internal Server Error, while docker push works just fine. This suggests to me that it isn’t just a display error in the UI, but a deeper issue.

The error can be “fixed” temporarily by one of the following actions:

  • using `s3cmd sync’ to sync the container registry contents from the S3-bucket GitLab uses to disk restores all tags for a while.
  • pushing an image to the affected image repostiroy with 'docker push` fixes a few tags, seemingly at random. This is also temporary.

Once “fixed”, docker push and pull work as expected, until the error recurrs a while later.

I am at a loss right now and hope that you guys will a few ideas. Thanks in advance.

2 Likes
2

So I’ve been working on this some more and analyzed the registry container’s log. As long as the registry works, the logs (for pulling an image from CLI) look like this:


{"auth_user_name":"redacted_user","correlation_id":"01GFTT9XTGQR2Y5YRF1T45X16Q","go_version":"go1.17.6","level":"info","msg":"authorized request","root_repo":"telemetry","time":"2022-10-20T13:46:12.690Z","vars_digest":"sha256:6ab6a6301bdeddbcd0e6193262b2c54740f4ea12640c81c951223cdd5ec842a3","vars_name":"telemetry/qtcreator/test","version":"v3.27.1-gitlab"}
{"auth_user_name":"redacted_user","correlation_id":"01GFTT9XTGQR2Y5YRF1T45X16Q","digest":"sha256:6ab6a6301bdeddbcd0e6193262b2c54740f4ea12640c81c951223cdd5ec842a3","go_version":"go1.17.6","level":"info","msg":"blob downloaded","redirect":true,"root_repo":"telemetry","size_bytes":893,"time":"2022-10-20T13:46:12.771Z","vars_digest":"sha256:6ab6a6301bdeddbcd0e6193262b2c54740f4ea12640c81c951223cdd5ec842a3","vars_name":"telemetry/qtcreator/test","version":"v3.27.1-gitlab"}
{"content_type":"application/octet-stream","correlation_id":"01GFTT9XTGQR2Y5YRF1T45X16Q","duration_ms":242,"host":"registry2.redacted.de","level":"info","method":"GET","msg":"access","proto":"HTTP/1.0","referrer":"","remote_addr":"10.3.1.176:42277","remote_ip":"10.3.1.176","status":307,"system":"http","time":"2022-10-20T13:46:12.931Z","ttfb_ms":82,"uri":"/v2/telemetry/qtcreator/test/blobs/sha256:6ab6a6301bdeddbcd0e6193262b2c54740f4ea12640c81c951223cdd5ec842a3","user_agent":"docker/20.10.14 go/go1.16.15 git-commit/87a90dc kernel/5.15.0-52-generic os/linux arch/amd64 UpstreamClient(Docker-Client/20.10.14 \\(linux\\))","written_bytes":0}

But once the registry enters “failure state”, the logs (again, pulling the same image from CLI) show an “Access Denied” error message:


{"auth_user_name":"redacted_user","correlation_id":"01GFTT6ADQ6A3DP4QQE1635SY4","go_version":"go1.17.6","level":"info","msg":"authorized request","root_repo":"telemetry","time":"2022-10-20T13:44:14.520Z","vars_name":"telemetry/qtcreator/test","vars_reference":"v1.0","version":"v3.27.1-gitlab"}
{"auth_user_name":"redacted_user","code":"UNKNOWN","correlation_id":"01GFTT6ADQ6A3DP4QQE1635SY4","detail":"AccessDenied: Access Denied.\n\tstatus code: 403, request id: 171FCAE0EC3F723D, host id: ","error":"unknown: unknown error","go_version":"go1.17.6","level":"error","msg":"unknown error","root_repo":"telemetry","time":"2022-10-20T13:44:14.540Z","vars_name":"telemetry/qtcreator/test","vars_reference":"v1.0","version":"v3.27.1-gitlab"}
{"content_type":"application/json","correlation_id":"01GFTT6ADQ6A3DP4QQE1635SY4","duration_ms":21,"host":"registry2.fza.redacted.de","level":"info","method":"GET","msg":"access","proto":"HTTP/1.0","referrer":"","remote_addr":"10.3.1.176:31974","remote_ip":"10.3.1.176","status":500,"system":"http","time":"2022-10-20T13:44:14.540Z","ttfb_ms":21,"uri":"/v2/telemetry/qtcreator/test/manifests/v1.0","user_agent":"docker/20.10.14 go/go1.16.15 git-commit/87a90dc kernel/5.15.0-52-generic os/linux arch/amd64 UpstreamClient(Docker-Client/20.10.14 \\(linux\\))","written_bytes":89}

The same error (“Access Denied”) also happens when I try to display the registry in the GUI in the browser, which makes me think this might be the root cause of my problems.

I am, however, not aware of any authentication or authorization necessary for fetching container registry image manifests. I’m not sure what to make of this, but any further suggestion on how to debug this futher is welcome.

2 Likes