I’m currently facing an issue with GitLab’s Docker Registry integration where the deletion of Docker images is attempted using tags instead of the digest, leading to failures. Specifically, when GitLab attempts to delete an image from the registry, it uses the image tag in the request, which results in an error. My setup involves a separate host for the Docker Registry, which is integrated with GitLab for managing Docker images. The registry is configured to allow deletions, and everything works flawlessly except for the image deletion part.
Here’s a sanitized version of my configurations:
GitLab (gitlab.rb) is set up to enable and point to the Docker Registry:
gitlab_rails['registry_enabled'] = true
gitlab_rails['registry_host'] = "registry.example.com"
gitlab_rails['registry_api_url'] = "https://registry.example.com"
Docker Registry is deployed using the following docker-compose.yml on a separate machine:
version: '3.8'
services:
registry:
image: registry:latest
ports:
- "5000:5000"
environment:
REGISTRY_STORAGE_FILESYSTEM_ROOTDIRECTORY: /var/lib/registry
REGISTRY_STORAGE_DELETE_ENABLED: "true"
volumes:
- /var/docker/gitlab-registry:/var/lib/registry
When GitLab tries to delete an image, the registry logs this request:
[28/Mar/2024:14:01:49 +0000] "DELETE /v2/some-awesome-repo/manifests/437 HTTP/1.0" 400 98 "" "GitLab/16.10.1-ee"
And reports an error:
level=error msg="response completed with error" err.code="digest invalid" err.message="provided digest did not match uploaded content" go.version=go1.20.8 http.request.host=registry.example.com http.request.id=1699e4fc-9aca-4f4f-a48e-32ebad839997 http.request.method=DELETE http.request.remoteaddr=192.168.0.105 http.request.uri="/v2/some-awesome-repo/manifests/437"
The error indicates a mismatch because the deletion request was made with a tag (437) instead of the expected digest.
Interestingly, I was able to delete the image manually using Postman with the correct digest:
DELETE https://registry.example.com/v2/some-awesome-repo/manifests/sha256:081d853456cd9a32a8fs2324asa7899a35f282bc9d9d3fabc4cfb4cce5e2624a
This issue seems to be specific to how GitLab constructs the deletion request, as manual deletion with the correct digest works as expected. I’m looking for guidance on how to resolve this, ensuring GitLab correctly uses digests instead of tags for image deletion requests. Any insights or workarounds from the community would be greatly appreciated.