Well, for those of you that know how to do this, I’m sure people like me look like we just don’t read and don’t pay attention but that’s not the case.
I’ve been reading gitlab article after article, blog posts, docs, anything I can find and so far, it’s a complete mystery why restoring the backup file is not working. Worse, the command keeps saying it cannot find the file, does not exist. That just makes zero sense :).
Hi,
The correct command is:
gitlab-backup restore BACKUP=1678378917_2023_03_09_15.9.2
as outlined here in the steps for restore: Redirecting... you also have to stop puma and sidekiq services before attempting restore, was that done? If you did a complete gitlab-ctl stop then that is wrong, and the restore won’t work. Some of the services have to be running, so all that is required is stop puma and sidekiq, leave remaining running, as it has to restore the database, etc. From the link above:
Also, it can only be restored to the same gitlab version. As we can see your backup is from 15.9.2 so it can only be restored to 15.9.2. Also if it is gitlab-ce it can only be restore to ce. You cannot restore EE to CE, or vice versa.
The hostname/domain FQDN of the new server doesn’t need to be the same. You can edit gitlab.rb and change it. Then run reconfigure to use the new settings.
If you installed an rpm or deb package or using package manager like yum/dnf/apt, then it is omnibus. The other versions are source installs which are compiled, or docker based.
Either way, it’s weird yours is not restoring, because every time I have done this it just works. And I haven’t done anything different to you. Maybe think about disabling selinux temporarily:
setenforce 0
then try restore. That will put it in permissive mode, so it will not block, but it will still log violations.
Otherwise you will have to check your /var/log/gitlab/gitlab-rails log files and post here. There must be something showing in the logs as to why it’s not working.
Here is an example from my test server now:
root@gitlab:~# cd /var/opt/gitlab/backups/
root@gitlab:/var/opt/gitlab/backups# ls -lha
total 564M
drwx------ 2 git root 4.0K Mar 9 16:56 .
drwxr-xr-x 23 root root 4.0K Mar 3 10:36 ..
-rw------- 1 git git 730K Mar 3 10:34 1677836057_2023_03_03_15.9.1_gitlab_backup.tar
-rw------- 1 git git 563M Mar 9 16:56 1678377364_2023_03_09_15.9.2_gitlab_backup.tar
so you can see similar filename format to yours, the file is owned by git:git and the permissions also the same as yours.
Now I stop puma and sidekiq:
root@gitlab:/var/opt/gitlab/backups# gitlab-ctl stop puma
ok: down: puma: 0s, normally up
root@gitlab:/var/opt/gitlab/backups# gitlab-ctl stop sidekiq
ok: down: sidekiq: 0s, normally up
the rest of Gitlab was left running. Now we start restore:
root@gitlab:/var/opt/gitlab/backups# gitlab-backup restore BACKUP=1678377364_2023_03_09_15.9.2
Transfering ownership of /var/opt/gitlab/gitlab-rails/shared/registry to git
2023-03-10 08:13:54 UTC -- Unpacking backup ...
2023-03-10 08:13:56 UTC -- Unpacking backup ... done
2023-03-10 08:13:56 UTC -- Restoring database ...
2023-03-10 08:13:56 UTC -- Be sure to stop Puma, Sidekiq, and any other process that
connects to the database before proceeding. For Omnibus
installs, see the following link for more information:
https://docs.gitlab.com/ee/raketasks/backup_restore.html#restore-for-omnibus-gitlab-installations
Before restoring the database, we will remove all existing
tables to avoid future upgrade problems. Be aware that if you have
custom tables in the GitLab database these tables and all data will be
removed.
Do you want to continue (yes/no)? yes
Removing all tables. Press `Ctrl-C` within 5 seconds to abort
2023-03-10 08:14:12 UTC -- Cleaning the database ...
2023-03-10 08:14:20 UTC -- done
Restoring PostgreSQL database gitlabhq_production ... ERROR: must be owner of extension pg_trgm
ERROR: must be owner of extension btree_gist
ERROR: must be owner of extension btree_gist
ERROR: must be owner of extension pg_trgm
...lots of postgres alter table output...
[DONE]
2023-03-10 08:14:36 UTC -- Restoring database ... done
2023-03-10 08:14:36 UTC -- Restoring repositories ...
{"command":"restore","gl_project_path":"test/test","level":"info","msg":"started restore","relative_path":"@hashed/d4/73/d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35.git","storage_name":"default","time":"2023-03-10T08:14:37.563Z"}
{"command":"restore","gl_project_path":"test/test","level":"info","msg":"completed restore","relative_path":"@hashed/d4/73/d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35.git","storage_name":"default","time":"2023-03-10T08:14:37.807Z"}
{"command":"restore","gl_project_path":"test/test.wiki","level":"info","msg":"started restore","relative_path":"@hashed/d4/73/d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35.wiki.git","storage_name":"default","time":"2023-03-10T08:14:37.858Z"}
{"command":"restore","gl_project_path":"test/test","level":"info","msg":"started restore","relative_path":"@hashed/d4/73/d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35.design.git","storage_name":"default","time":"2023-03-10T08:14:37.865Z"}
{"command":"restore","error":"manager: repository skipped: restore bundle: filesystem sink: get reader for \"@hashed/d4/73/d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35.design.bundle\": doesn't exist","gl_project_path":"test/test","level":"warning","msg":"skipped restore","relative_path":"@hashed/d4/73/d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35.design.git","storage_name":"default","time":"2023-03-10T08:14:37.883Z"}
{"command":"restore","gl_project_path":"docker/docker-registry","level":"info","msg":"started restore","relative_path":"@hashed/ef/2d/ef2d127de37b942baad06145e54b0c619a1f22327b2ebbcfbec78f5564afe39d.git","storage_name":"default","time":"2023-03-10T08:14:37.883Z"}
{"command":"restore","gl_project_path":"test/test.wiki","level":"info","msg":"completed restore","relative_path":"@hashed/d4/73/d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35.wiki.git","storage_name":"default","time":"2023-03-10T08:14:37.999Z"}
{"command":"restore","gl_project_path":"docker/docker-registry.wiki","level":"info","msg":"started restore","relative_path":"@hashed/ef/2d/ef2d127de37b942baad06145e54b0c619a1f22327b2ebbcfbec78f5564afe39d.wiki.git","storage_name":"default","time":"2023-03-10T08:14:37.999Z"}
{"command":"restore","error":"manager: repository skipped: restore bundle: filesystem sink: get reader for \"@hashed/ef/2d/ef2d127de37b942baad06145e54b0c619a1f22327b2ebbcfbec78f5564afe39d.wiki.bundle\": doesn't exist","gl_project_path":"docker/docker-registry.wiki","level":"warning","msg":"skipped restore","relative_path":"@hashed/ef/2d/ef2d127de37b942baad06145e54b0c619a1f22327b2ebbcfbec78f5564afe39d.wiki.git","storage_name":"default","time":"2023-03-10T08:14:38.021Z"}
{"command":"restore","gl_project_path":"docker/docker-registry","level":"info","msg":"started restore","relative_path":"@hashed/ef/2d/ef2d127de37b942baad06145e54b0c619a1f22327b2ebbcfbec78f5564afe39d.design.git","storage_name":"default","time":"2023-03-10T08:14:38.021Z"}
{"command":"restore","error":"manager: repository skipped: restore bundle: filesystem sink: get reader for \"@hashed/ef/2d/ef2d127de37b942baad06145e54b0c619a1f22327b2ebbcfbec78f5564afe39d.design.bundle\": doesn't exist","gl_project_path":"docker/docker-registry","level":"warning","msg":"skipped restore","relative_path":"@hashed/ef/2d/ef2d127de37b942baad06145e54b0c619a1f22327b2ebbcfbec78f5564afe39d.design.git","storage_name":"default","time":"2023-03-10T08:14:38.038Z"}
{"command":"restore","gl_project_path":"docker/docker-registry","level":"info","msg":"completed restore","relative_path":"@hashed/ef/2d/ef2d127de37b942baad06145e54b0c619a1f22327b2ebbcfbec78f5564afe39d.git","storage_name":"default","time":"2023-03-10T08:14:38.041Z"}
2023-03-10 08:14:38 UTC -- Restoring repositories ... done
2023-03-10 08:14:38 UTC -- Restoring uploads ...
2023-03-10 08:14:38 UTC -- Restoring uploads ... done
2023-03-10 08:14:38 UTC -- Restoring builds ...
2023-03-10 08:14:38 UTC -- Restoring builds ... done
2023-03-10 08:14:38 UTC -- Restoring artifacts ...
2023-03-10 08:14:38 UTC -- Restoring artifacts ... done
2023-03-10 08:14:38 UTC -- Restoring pages ...
2023-03-10 08:14:38 UTC -- Restoring pages ... done
2023-03-10 08:14:38 UTC -- Restoring lfs objects ...
2023-03-10 08:14:38 UTC -- Restoring lfs objects ... done
2023-03-10 08:14:38 UTC -- Restoring terraform states ...
2023-03-10 08:14:38 UTC -- Restoring terraform states ... done
2023-03-10 08:14:38 UTC -- Restoring container registry images ...
2023-03-10 08:14:44 UTC -- Restoring container registry images ... done
2023-03-10 08:14:44 UTC -- Restoring packages ...
2023-03-10 08:14:44 UTC -- Restoring packages ... done
This task will now rebuild the authorized_keys file.
You will lose any data stored in the authorized_keys file.
Do you want to continue (yes/no)? yes
2023-03-10 08:15:14 UTC -- Deleting tar staging files ...
2023-03-10 08:15:14 UTC -- Cleaning up /var/opt/gitlab/backups/backup_information.yml
2023-03-10 08:15:14 UTC -- Cleaning up /var/opt/gitlab/backups/db
2023-03-10 08:15:14 UTC -- Cleaning up /var/opt/gitlab/backups/repositories
2023-03-10 08:15:14 UTC -- Cleaning up /var/opt/gitlab/backups/uploads.tar.gz
2023-03-10 08:15:14 UTC -- Cleaning up /var/opt/gitlab/backups/builds.tar.gz
2023-03-10 08:15:14 UTC -- Cleaning up /var/opt/gitlab/backups/artifacts.tar.gz
2023-03-10 08:15:14 UTC -- Cleaning up /var/opt/gitlab/backups/pages.tar.gz
2023-03-10 08:15:14 UTC -- Cleaning up /var/opt/gitlab/backups/lfs.tar.gz
2023-03-10 08:15:14 UTC -- Cleaning up /var/opt/gitlab/backups/terraform_state.tar.gz
2023-03-10 08:15:14 UTC -- Cleaning up /var/opt/gitlab/backups/registry.tar.gz
2023-03-10 08:15:14 UTC -- Cleaning up /var/opt/gitlab/backups/packages.tar.gz
2023-03-10 08:15:14 UTC -- Deleting tar staging files ... done
2023-03-10 08:15:14 UTC -- Deleting backups/tmp ...
2023-03-10 08:15:14 UTC -- Deleting backups/tmp ... done
2023-03-10 08:15:14 UTC -- Warning: Your gitlab.rb and gitlab-secrets.json files contain sensitive data
and are not included in this backup. You will need to restore these files manually.
2023-03-10 08:15:14 UTC -- Restore task is done.
2023-03-10 09:15:14 +0100 -- Deleting backup and restore lock file
Transfering ownership of /var/opt/gitlab/gitlab-rails/shared/registry to registry
and my system is restored. The only steps to be done now are:
gitlab-ctl reconfigure
gitlab-ctl restart
the postgres errors about btree can be ignored. They are not important. I restored the 15.9.2 backup because that is the version of Gitlab installed. It was also gitlab-ce, omnibus, so providing you have all these requirements, and the file is in /var/opt/gitlab/backups, owned by git:git, and you didn’t stop all gitlab services, then there’s no reason for it to fail. But we would need more info from your environment to ensure this all matches.
Hi,
I followed the directions you’ve given and in the article you shared.
The versions are identical other than centos 7.9 to Rocky 9.
The backup was done this way, “sudo gitlab-backup create”.
I checked the tarball and it’s ok, it’s not broken or anything.
# sudo gitlab-ctl status
run: alertmanager: (pid 121106) 113s; run: log: (pid 929) 75982s
run: gitaly: (pid 121119) 113s; run: log: (pid 936) 75982s
run: gitlab-exporter: (pid 121140) 112s; run: log: (pid 922) 75982s
run: gitlab-kas: (pid 121142) 112s; run: log: (pid 919) 75982s
run: gitlab-workhorse: (pid 121158) 111s; run: log: (pid 925) 75982s
run: logrotate: (pid 121170) 111s; run: log: (pid 924) 75982s
run: nginx: (pid 121176) 110s; run: log: (pid 927) 75982s
run: node-exporter: (pid 121188) 110s; run: log: (pid 918) 75982s
run: postgres-exporter: (pid 121197) 110s; run: log: (pid 920) 75982s
run: postgresql: (pid 121204) 109s; run: log: (pid 931) 75982s
run: prometheus: (pid 121213) 109s; run: log: (pid 938) 75982s
run: puma: (pid 121227) 108s; run: log: (pid 928) 75982s
run: redis: (pid 121232) 108s; run: log: (pid 934) 75982s
run: redis-exporter: (pid 121241) 107s; run: log: (pid 932) 75982s
run: sidekiq: (pid 121248) 107s; run: log: (pid 926) 75982s
# sudo gitlab-ctl stop puma
ok: down: puma: 1s, normally up
# sudo gitlab-ctl stop sidekiq
ok: down: sidekiq: 0s, normally up
# sudo gitlab-ctl status
run: alertmanager: (pid 121106) 135s; run: log: (pid 929) 76004s
run: gitaly: (pid 121119) 135s; run: log: (pid 936) 76004s
run: gitlab-exporter: (pid 121140) 134s; run: log: (pid 922) 76004s
run: gitlab-kas: (pid 121142) 134s; run: log: (pid 919) 76004s
run: gitlab-workhorse: (pid 121158) 133s; run: log: (pid 925) 76004s
run: logrotate: (pid 121170) 133s; run: log: (pid 924) 76004s
run: nginx: (pid 121176) 132s; run: log: (pid 927) 76004s
run: node-exporter: (pid 121188) 132s; run: log: (pid 918) 76004s
run: postgres-exporter: (pid 121197) 132s; run: log: (pid 920) 76004s
run: postgresql: (pid 121204) 131s; run: log: (pid 931) 76004s
run: prometheus: (pid 121213) 131s; run: log: (pid 938) 76004s
down: puma: 13s, normally up; run: log: (pid 928) 76004s
run: redis: (pid 121232) 130s; run: log: (pid 934) 76004s
run: redis-exporter: (pid 121241) 129s; run: log: (pid 932) 76004s
down: sidekiq: 5s, normally up; run: log: (pid 926) 76004s
# ll
total 713060
-rw------- 1 git git 730173440 Mar 9 09:22 1678378917_2023_03_09_15.9.2_gitlab_backup.tar
# gitlab-backup restore BACKUP=1678378917_2023_03_09_15.9.2
2023-03-10 13:48:03 UTC -- Unpacking backup ...
2023-03-10 13:48:03 UTC -- Unpacking backup failed
2023-03-10 06:48:03 -0700 -- Deleting backup and restore lock file
Going from;
to;
Did I miss something?
Can you try:
setenforce 0
just to rule out if selinux is a problem or not and try the restore again? This will put it in permissive mode, and if it works, we can check /var/log/audit/audit.log for more details.
I’ve restored my Debian install to Rocky 8 before now with selinux enabled, and that has worked fine, so curious if it’s something with EL9 that’s possibly an issue.
Sorry, forgot to mention that I tried that.
Here it is again just to be safe.
# setenforce 0
setenforce: SELinux is disabled
# gitlab-backup restore BACKUP=1678378917_2023_03_09_15.9.2
2023-03-10 15:05:45 UTC -- Unpacking backup ...
2023-03-10 15:05:45 UTC -- Unpacking backup failed
2023-03-10 08:05:45 -0700 -- Deleting backup and restore lock file
There isn’t anything interesting in the logs that I’ve not shared.
The only difference is a slight version diff in PostgreSQL.
Weird uh?
So the postgres versions will also need to be the same, if your old install has postgres 12 then you can upgrade it manually, make the backup, and restore to the new install. I believe the new install would have defaulted to postgres 13.
gitlab-ctl pg-upgrade
Check your versions installed:
root@gitlab:~# gitlab-psql --version
psql (PostgreSQL) 13.8
on old and new install to make sure they are the same.
On the old server,
PostgreSQL (main) 12.12
On the new server,
PostgreSQL (main) 13.8
I’ll take a look at upgrading but I upgraded the old system before doing the backup.
I think that’s as high as it goes on centos 7.9 perhaps.
What is happening that I cannot get this done? It’s been days now and it’s becoming a bit critical as there are a bunch of network changing going on and I have to move the old server data to this new server.
Is there anyone here that can help me? I’ve been at this since last week and badly need to move this server over.
You will have to upgrade postgres on the old server, then make a backup. You cannot restore postgres 12 to postgres 13.
Unfortunately, the lack of information from your side, from the log files, etc means it’s virtually impossible to help. I showed you a working example. Obviously other than the postgres issue, there is a problem with your backup. You could check the md5sum of this file on the old server, and compare it with the one on the new server and see if it’s the same. If not, then it means when it was copied over, it is corrupted and could explain the inability to unpack it. Or perhaps there wasn’t enough free disk space on your old server to make a proper complete backup, or perhaps there wasn’t enough space on the new server to unpack it, and it failed. But I don’t know if these are possibilities or sources of the problem, because we don’t know for sure if your backup worked properly, or if there is enough disk space.
It might be critical, but unfortunately, due to the lack of information it’s virtually impossible to help. I suggest you go through all the log files carefully and try to find anything that might hint at why it failed. If there is nothing, then how do you expect anyone to help you fix it? We can only guess where the problem might be.
You do realise this is free support? There are no guarantees of assistance or even resolving your problem. It’s a best endeavour from kind people in the community that give up their time to help you. Now if you paid for support, then you would have a right to get disappointed if the problem isn’t getting solved in a timely manner.
I understand. I would love to provide more information but there isn’t any to provide. There’s nothing in the logs that I’ve not shared already. I overlooked the db version so I’ll work on that next and see where I get.
So far, there is no upgrade path on centos 7.9 and no downgrade path on rocky 9.
I’m surprised there would be such large differences between the versions. I’ve been using mysql for over ten years and it’s always painless to move from one to another.
Once you sort out postgres so that both old and new installs are on 13, then we can continue with the backup.
So, after you have upgraded, on the old server run the gitlab-backup command as per my example. Then, for the file that was created in /var/opt/gitlab/backups, do:
md5sum 1678378917_2023_03_09_15.9.2_gitlab_backup.tar
obviously change the above filename to the new backup file that was just created. Then, copy the backup file to the new server, place the file in /var/opt/gitlab/backups and then check the md5sum again. The result should be exactly the same from the old server. This means it was copied over correctly, and file integrity is OK. Any difference, and it means it has failed to copy over properly.
Then try the restore again. Make sure of course, that the server has plenty of space available to extract the backup, especially for the entire /var/opt/gitlab where all the repositories, etc will get extracted to.
Just for the hell of it, I tried;
Old server;
Moved the file over, then;
2023-03-13 17:56:29 UTC – Unpacking backup …
2023-03-13 17:56:29 UTC – Unpacking backup failed
2023-03-13 10:56:29 -0700 – Deleting backup and restore lock file
I’ve looked on the net, I don’t see a downgrade or upgrade path as explained above.
Not sure what to try next.
You don’t say whether you checked the md5sum or not to verify whether it was copied across successfully or not.
I have no further ideas, if there’s nothing in the logs, and both are now on postgres 13, then there is obviously something broken with your system. I’ve recovered mine multiple times over the years, be it restoring to a new install on Debian, or even on Rocky Linux (but version 8.7 not 9), and it worked every time.
The fact it’s not unpacking your backup would mean there is something wrong with the tar file created in that backup process or it has not been copied over properly and the md5sums didn’t match.
Sorry, I forgot to post that;
After creating the backup;
4438d37619fb395542e75af26e99c816 1678728896_2023_03_13_15.9.2_gitlab_backup.tar
After moving it to the new server;
4438d37619fb395542e75af26e99c816 1678728896_2023_03_13_15.9.2_gitlab_backup.tar
I’m not doing anything unusual and both gitlab instances seem to be running perfectly so it’s a mystery that I cannot get this done.
Maybe you can make a test server with Rocky 8 instead and see if you can get it restored there. Since Gitlab don’t yet provide an official EL9 package yet.
That might be the only way. I"ll take a look at that.
I just finished installing a rocky 8 and 15.9.3 only to find PostgreSQL 13.8.
The old Centos updated to 15.9.3 has 12.12.
There’s just no winning this. Why is the database version so incredibly important? I’ve never had this problem using mysql.