Restoring a backup of Omnibus to a completely clean system, and it fails during the Restoring container registry images because of a missing file /var/opt/gitlab/backups/registry.tar.gz
The file doesn’t exist on the backup-source or the destination computer.
Why is it looking for this file, or why is it missing?
2022-01-13 21:21:27 +0000 -- Restoring container registry images ...
rake aborted!
Errno::ENOENT: No such file or directory - /var/opt/gitlab/backups/registry.tar.gz
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/files.rb:76:in `run_pipeline!'
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/files.rb:44:in `restore'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/backup.rake:245:in `block (4 levels) in <top (required)>'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/backup.rake:79:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Tasks: TOP => gitlab:backup:registry:restore
(See full trace by running task with --trace)
Transfering ownership of /var/opt/gitlab/gitlab-rails/shared/registry to registry
When you restore, you have to copy all files to the /var/opt/gitlab/backups directory. Therefore, if you didn’t copy back the registry.tar.gz then it’s not going to restore it.
It all depends on how your backups were done, as per the docs: Back up and restore GitLab | GitLab you can choose what the backups include and exclude. We would need to know more about what commands you ran to make the backups in the first place. But also, if you want to restore, you have to make sure all required files are in the correct location for it to work. Obviously a backup made the file in /var/opt/gitlab/backups, but it hasn’t been put back for the restore process which you have to do manually yourself.
The link above also has the steps required for restoring as well as backup. But once you let us know what commands you used for making the backup, it will be obvious where the problem lies.
Backup was taken with no-frills /usr/bin/gitlab-backup create
The only backup-related entry in gitlab.rb is gitlab_rails['backup_keep_time']
I don’t have a SKIP environment variable defined.
There is never a registry.tar.gz file in /var/opt/gitlab/backups.
Seems strange. Did the remainder of the restore work? I don’t see anything else in your initial post on this. Perhaps the restore completed fine, but just reported an error for this particular file not existing. So perhaps the problem isn’t that big at all if the remainder of the server is restored.
That aside, you should be running the latest Gitlab, 12.9.2 and a lot of 13.x versions had serious vulnerabilities and servers were compromised because of it. Crypto miners were running and using all the CPU resources and causing Gitlab to die. So assuming the restore worked fine without the registry file, I would suggest upgrading asap to the latest version following the Gitlab docs upgrade guide and following the upgrade path to make sure certain important versions inbetween your current and latest version aren’t missed.
I know we need to upgrade - but for various reasons I’m unable currently to upgrade the source system - but plan to upgrade the restored system as soon as it restores successfully.
I don’t think the restore worked. It returned to the prompt immediately after the Transfering ownership message - no “completed successfully” message (though I’m not entirely sure what a successful restore looks like).
I’m going to take a fresh, full backup and try restoring that.
Thanks
The service is definitely not restored - I get a 500 error trying to login.
OK, let us know what happens after full fresh backup. Also make sure you have the gitlab.rb configuration file and gitlab_secrets.json also as per the gitlab backup/restore docs.