Hello all,
Trying to clean up some extra room on our Gitlab server as disk space is getting quite low.
I have narrowed the bulk of the data to /var/opt/gitlab/gitlab-rails/shared/registry/docker/registry/v2/blobs/sha256 but not sure what this folder structure represents.
There is currently 61G of data in this folder alone. I assume it is directly related to the registry. However, the only project that is using the registry only appears to be using around 4GB of data.
Is someone able to shed some light as to what is going on here?
Disk usage:
Registry usage:
1 Like
Hi @cybersecfun this is all related to the Container Registry. The issue is related to how images are deleted.
There are two pieces to how old images are removed…
The first is untagging images in the registry. It’s important to note that removing a tag does not delete the image. You still have to run garbage collection to remove the actual blobs. We currently support this via API, so project maintainers can untag images in bulk based on different criteria. Here is the documentation for the API:https://docs.gitlab.com/ce/api/container_registry.html#delete-repository-tags-in-bulk
Once untagged, you can run garbage collection and remove the blobs from storage. Here is the documentation on how to run garbage collection.https://docs.gitlab.com/omnibus/maintenance/#container-registry-garbage-collection
There are several ways to run garbage collection, including on a schedule. It’s not policies and automation, but we are working towards that feature set now.
I’ve also seen a couple of examples from other users who have created scripts for this, in case you are looking for some automation inspiration. (Both scripts are referenced here)
Hopefully this helps you solve your problem!
2 Likes
Thank you @trizzi for the feedback.
I have run through the recommended steps but seem to be in the same boat.
We have removed a few of the tags that we no longer need, ran garbage collection and still seem to have storage utilization much higher than what we would expect.
Again, reviewing the registry for this particular project the total size appears to be 5.38157005608082 GB. This is reported in both the GUI and API.
However, pulling the storage stats from the filesystem, the registry is using 96GB.
I am not sure how to correlate the file structure with a project’s registry but I am certain that the project I am referencing would be the culprit as it is the only one that is utilizing automated builds etc.
Thanks for any additional info you may have.
No one ever seems to know why there’s this huge discrepancy. Did you find a solution? I have the unsatisfying feeling that there is a lot of legacy data which has not been properly removed by the GitLab upgrade scripts.
