All repos empty after backup & restore

I’m in the process of upgrading our gitlab-ce installation (v11.10.4) which is currently running on Debian 9. I also want to upgrade the OS at the same time, I decide that the best way to do this was first duplicate what I have on to a 2nd server, complete the gitlab-ce upgrades to the latest version and then move the upgraded gitlab-ce installation to the Debian 11 server.

On our original server I stopped sidekiq & unicore services and ran gitlab-rake gitlab:backup:create SKIP=repositories. I grab the backup file, the required files from /etc/gitlab

I then copied the repositories to a mounted vhd, waited for that to complete.
Mounted the vhd on the new Debain 9 server under the same directory as is configured on the original server.
Restored the backup, copied the files to /etc/gitlab and ran gitlab-ctl reconfigure and logged into gitlab.

Upon browsing to the repositories they are all empty. It just gives me the option to import or create an empty repository. If I create a new project that project repo gets written to the same place as the other repositories (The mounted vhd) and I only have 1 path configured in gitlab.rb. If I browse the activity that is all correct, the commits are all correct. There are just no files there.

I’ve checked the permissions on the directory, I’ve created additional backups and repeated the process. I’ve even tried creating an empty repository and copying over the contents.

None of these have worked.

1 Like

Did you figure this out? I’m restoring a Community Edition from an older version of Red Hat to a newer one. My repos are missing code, but the repos exist. Both are on v15.11.9.

I think it was somewhere between 11.10 and current versions that storage change to hashed format, that might explain why repositories can’t be found by a new GitLab given a disk written by an old GitLab.
I don’t know of any easy ways to recover from that.

1 Like

The backup and restore commands state that the source and destination GitLab versions be identical. My servers are v15.11.9. Thing is, I used the identical steps to migrate a Premium Edition of GL a couple months ago and the steps worked. I didn’t run into this problem on that other one. This instance though is Community, and I don’t have support on it. :frowning:

As you didn’t write much I was mainly reacting to @drifter104 's post (no idea why I didn’t see that when it was new), and doing the backup with SKIP=repositories and copying the repositories like described sounds like a recipe for trouble.

I can see you actually did write that both versions in your case was 15.11.9, then my response doesn’t apply, but to defend myself you probably shouldn’t have written in an old thread that contained a description of a method different from yours.

I have no idea what happened on your servers, did the restore throw any unusual messages?

1 Like

No worries. I replied to this old message because the outcome was identical.

I didn’t see anything unusual in the output of my restoration.

Sorry for not seeing this before.

I got this working in the end by doing things slightly differently.

I created a duplicate server (migration server) running the same version of gitlab-ce as my production server.
Attached a blank VHD to my production server and cp my files to the new vhd.
Did a backup of my production server with SKIP=repositories
Moved that VHD to my migration server.

Then on the migration server I did several upgrade, looking at the release notes for each version to see what minimum version the next one needed. At the time I needed to do about 8 upgrades.

Then once I was running the latest version, I built a Debain 11 server and did the reverse of the above.
Attached my vhd to the new production server, did a backup of my migration server and then restored the backup to the new production server.

If you’re still going to do it make sure you pay special attention to the version that switches to hashed storage, there are several behind the scenes tasks that you just have to wait on.

I did the process twice. The first time the change to hashed storage just didn’t work during the upgrade process. The 2nd time while do the upgrades process I created a vm snapshot before each upgrade step so I could roll back easily.

1 Like

Thank you.

This kinda overlaps what I’ve done. Existing CE is on Red Hat 7, new CE is on RH8. The versions of GitLab are identical, v15.11.9. I’m not crossing the hashed storage version boundary at all.

Having the same Problem with a 15.11.9 . I Updated to 16.x and the Repositorys were shown as empty. Trying to restore the Backup with 15.11.9 but the stying empty. They are inside the backup. Realy wird.

I’m going to update to 15.11.13 during the next maintenance period and try the backup/restore process again. :crossed_fingers: