Different between registry-garbage-collect and registry-garbage-collect -m

General
,
1

Hi!

I read the documentation about Container Registry garbage collection. In the documentation two commands are mentioned:

  • registry-garbage-collect
  • registry-garbage-collect -m

I think I understand what the command registry-garbage-collect -m does. This will delete all blobs/layers that are not used for an image tag. This is also well described in the documentation.

But what is not clear to me: What does registry-garbage-collect do? The documentation only describes the process (1. stop registry, 2. garbage collector runs, 3. start registry), but not what it does exactly and what result I can expect. Is there a better explanation somewhere of what registry-garbage-collect does and how this command differs from registry-garbage-collect -m?

Thanks,
Markus

1 Like
2

I have tried to understand what registry-garbage-collect does.

  1. I built a 2GB image and pushed it into the registry
  2. Have deleted the image tag in GitLab Ui
  3. Have executed registry-garbage-collect: Disk space usage has not changed.
  4. Have executed registry-garbage-collect -m: Disk space has been released.

It is still not clear to me what registry-garbage-collect does.

3

the -m flag is the short form for --delete-manifests and it is currently a bit too aggressive

4

Thanks for your answer @NeroBurner. The conclusion for me is:

  • registry-garbage-collect -m should not be executed under certain circumstances*, because there has been a bug[1][2] for at least 11 months (the bug is not mentioned in the documentation).
  • registry-garbage-collect can be executed safely. But for me there is no real description of what this command does. According to my tests nothing is deleted.

As I understand it, there is currently not a single command with which I can safely delete untagged images. I feel a bit uncomfortable knowing that I need more and more TB of disk space but can’t do anything about it. This makes the Docker Registry, what can I say, not really recommendable to me :slight_smile:.

It’s like having a file system that can’t delete anything and just grows and grows with every write operation.

* You must not use multi-arch manifests. But as an admin you often don’t know if a user is using it.

5

@mritzmann without the -m flag, the garbage collector will only delete unused layers. As an example, if you delete a manifest which is the only one referencing layers A and B, those layers will be deleted.

6

There is indeed a bug affecting manifest lists. Due to technical limitations related to the registry storage layout, it’s not feasible to handle these during garbage collection (more about this here). We’re working on moving manifests from the registry filesystem into a SQL database to overcome this and other limitations.

7

For clarification: The bug with -m only has an effect if you create multi-arch manifests. For example a manifest for ubuntu:latest which links to amd64/ubuntu:latest and arm64v8/ubuntu:latest

This kind of manifest is wrongly deleted currently. The manifests for amd64/ubuntu:latest and arm64v8/ubuntu:latest are still intact

8

For multi-arch manifests, you should be able to mitigate this issue by pushing the manifests for a single arch to the same path as the multi-arch manifest.

So instead of ubuntu:latest which links to amd64/ubuntu:latest and arm64v8/ubuntu:latest do this: ubuntu:latest which links to ubuntu:latest-amd64 and ubuntu:latest-arm64v8

1 Like
9

Ok but how do you delete a specific manifest in a common GitLab usage (ie, through the UI) ?

2 Likes
10

Did you find any solution?
how do you clean for today?

11

I periodically run registry-garbage-collect -m because in my case nobody use multi-arch images at the moment. Since then, however, I have no longer dealt with the topic.

1 Like