Hello everyone,
We started to use the Omnibus docker registry but forgot to setup the registry garbage collector, now the command didn’t finish because of the huge quantity of files I guess. Is there a way to resolve this?
Environment info:
- Gitlab omnibus on latest version
- Registry storage: AWS S3
Logs of last run:
gitlab-ctl registry-garbage-collect:
INFO[29302] blob eligible for deletion digest="sha256:fffff5d5b8a42652af1ce634dec6d9189f4f353f3e8dbf20045cdeb2a99a96de" environment=production go_version=go1.14.7 instance_id=985835[77/1856]
136-9e82-45e7e2708b07 service=registry size_bytes=10214922
INFO[29302] mark stage complete blobs_marked=441935 blobs_to_delete=41680 duration_s=29302.057134891 environment=production go_version=go1.14.7 instance_id=98583552-00bd-4136-9e82-45$7e2708b07 manifests_to_delete=0 service=registry storage_use_estimate_bytes=785001053714
INFO[29302] starting sweep stage environment=production go_version=go1.14.7 instance_id=98583552-00bd-4136-9e82-45e7e2708b07 service=registry
INFO[29302] deleting blobs count=41680 environment=production go_version=go1.14.7 instance_id=98583552-00bd-4136-9e82-45e7e2708b07 service=registry
WARN[29317] blobs partially deleted count=4000 duration_s=14.99178148 environment=production go_version=go1.14.7 instance_id=98583552-00bd-4136-9e82-45e7e2708b07 service=registry
failed to garbage collect: failed to delete blobs: SlowDown: Please reduce your request rate.
status code: 503, request id: 03FB3A0B36F46333, host id: suS1LEE6buQrO/LOgqNNCx8yeD1u34ZSENUM24p4PTRnzQbgMSr++hXmOKrmjhI0nCIuk7OLtok=
– gitlab-ctl registry-garbage-collect -m:
79304ea5a9 manifests_to_delete=50791 service=registry storage_use_estimate_bytes=3578150785668
INFO[24292] starting sweep stage environment=production go_version=go1.14.7 instance_id=927da05e-38ec-4fdc-a331-9279304ea5a9 service=registry
INFO[24292] deleting manifests in batches batch_count=507 batch_max_size=100 environment=production go_version=go1.14.7 instance_id=927da05e-38ec-4fdc-a331-9279304ea5a9 service=registry
INFO[24292] preparing batch batch_number=1 batch_total=507 environment=production go_version=go1.14.7 instance_id=927da05e-38ec-4fdc-a331-9279304ea5a9 service=registry
INFO[24292] deleting batch batch_number=1 environment=production go_version=go1.14.7 instance_id=927da05e-38ec-4fdc-a331-9279304ea5a9 manifests=100 service=registry tags=17600 total=17700
WARN[24304] batch partially deleted batch_number=1 count=0 duration_s=11.152784904 environment=production go_version=go1.14.7 instance_id=927da05e-38ec-4fdc-a331-9279304ea5a9 service=registry
failed to garbage collect: failed to delete manifests: SlowDown: Please reduce your request rate.