Backup of registry only

Hi,
There are some similar question around. I shall move Gitlab (omnibus) to another server, my registry is huge and I can’t backup it up, it hangs up forever.
I succeed to do backup/restore of everything else, database, repositories with command:

$ gitlab-backup create SKIP=builds,artifacts,registry STRATEGY=copy

As far as I can see, this backup process creates several directories using the same structure as within directory:

$ ls /var/opt/gitlab/gitlab-rails/shared/
artifacts    cache  dependency_proxy  external-diffs  lfs.1485201713  lfs-objects  packages  pages  registry  terraform_state  tmp

I was hoping that later syncing directory /var/opt/gitlab/gitlab-rails/shared/registry would be good enough. Actually, I have to use external mount volume as it use a lot of space.
My related /etc/gitlab/gitlab.rb looks like (I tried several variations):

registry['debug_addr'] = "localhost:5001"
registry_external_url 'https://git-registry.domain.com'
registry_nginx['ssl_certificate']     = "/etc/gitlab/ssl/domain.com.crt"
registry_nginx['ssl_certificate_key'] = "/etc/gitlab/ssl/domain.com.key"
registry['registry_http_addr'] = '0.0.0.0:5000'
gitlab_rails['registry_path'] = "/mnt/gitlab-registry/BACKUP/registry/"

I can see in gitlab only the names of the docker images but when I click on them, no tags at all.

When I try to use localhost:5001 for debug, looks like permissions are not correct:

$ curl localhost:5001/debug/health
{"storagedriver_filesystem":"filesystem: stat /mnt/gitlab-registry/BACKUP/registry: permission denied"}

Now, permissions are the same like on the old instance:

$ ll /mnt/gitlab-registry/BACKUP/registry
total 8.0K
drwxr-xr-x 3 registry registry 4.0K Jan 17  2018 docker

And inside of the docker all permissions are given to registry.registry

Does anyone have an idea what am I missing here.

I tried to do alternative move of the mounted disc to separate VM with docker registry instance where just putting all registry directory on the right path was enough that registry service is aware of it, it was possible to use docker pull and other commands.

However, alternative is using external registry, but here manual states more about external domain name, not really as external registry - if I’m not wrong about it.

Thanks in advance.

There must be some data related to the registry (I imagine the tags you’re missing are there) in the database, and to avoid too much clutter building up in the database, that must be cleaned somewhere in the process.

I don’t know when that cleanup is done. If it’s when the backup is taken, you’ve pretty much lost, if it’s when the backup is restored, you can try syncing before, then the process will find the data and maybe leave references to it in the backup.

Note: This is not a part of the backup/restore-process I have touched, so I’m just guessing.

Hi grove,
thanks for reply, actually I figure it out, the permissions over registry files were problem.
Strange because first 2-3 top directories looked fine, and directory where manifests for repositories lives had owner root instead registry:registry. Not clear how did change happened. I didn’t even need to reconfigure or restart gitlab.

I’ve got this information enabling registry debug in /etc/gitlab/gitlab.rb:
registry['debug_addr'] = "localhost:5001"
Checking on the same server with:
curl localhost:5001/debug/health
got permission denied , then I went to check permissions in more details.

So, for a reference, basically it is enough to have in gitlab.rb other (mounted or else pointed directory) like:
gitlab_rails['registry_path'] = "/mnt/gitlab-registry/"
and make sure that first directory has permission 770, git:registry
and for all subdirectories owner must be registry:registry

So, it looks good now, thanks.