Repos are empty after migrating GitLab Community Edition v15.11.9 from Red Hat 7.9 to Red Hat 8.8

I inherited a Red Hat 7 (“RH7” / “RH8”) VM running GitLab (“GL”) Community Edition (“CE”). It’s been updated regularly. It’s currently on v15.11.9. My company is getting off of RH7 wherever possible, and this one is no exception. I stood up a new RH8 VM, installed the identical version of GL using the Omnibus directions.

I’m using this command to create the backup on the source VM

gitlab-backup create STRATEGY=copy SKIP=tar --trace

, which yields 1.2 TB of data in “/var/opt/gitlab/backups”, which is a symlink to another mount point.

On the destination VM, RH8, I run these commands

# SSH keys, gitlab.rb, and gitlab secrets JSON file are already migrated
gitlab-ctl status
gitlab-ctl stop # Ensure all connections to the db are shutdown
export DISABLE_DATABASE_ENVIRONMENT_CHECK=1 # I've run with this omitted as well
export GITLAB_ASSUME_YES=1 # I've run with this omitted as well
gitlab-ctl start postgresql # Start just the postgres server
gitlab-rake db:drop # Drop the existing database
gitlab-ctl reconfigure # Have omnibus create a fresh database
gitlab-ctl restart # Bring the fresh instance back online
gitlab-ctl reconfigure
cd /var/opt/gitlab/backups/ # backups were rsynced to this location. No singular TAR file, just the data.
gitlab-ctl status
gitlab-ctl stop puma # Instructions say to stop this service
gitlab-ctl stop sidekiq # Instructions say to stop this service
gitlab-ctl status # Confirm the services are stopped
gitlab-backup restore --trace force=yes # This restore takes 19 hours
gitlab-ctl reconfigure
gitlab-ctl restart
gitlab-ctl status

and the restore seems to finish correctly; everything in the web interface seems good. In the Admin interface, the number of projects and users is accurate. Also in the admin interface (“Overview” → “Projects” → sorted by “Largest repository”), the sizes for the repos are correct. However, as the title states, the repositories are empty after the restoration finishes. I get the usual message for an empty repo:

The repository for this project is empty

You can get started by cloning the repository or start adding files to it with one of the following options.

This post All repos empty after backup & restore is close to my issues¹, but I’m not crossing a hashed storage repo boundary like that post’s author.

Lastly, I used the above steps to migrate an instance of GitLab Premium Edition - which was also a very recent v15 version - from RH7 to RH8 and it worked.

I appreciate any and all assistance.

¹Something else that happened during the initial backup attempt was that all container images and related tags disappeared. There weren’t many; users re-uploaded images and re-tagged them.

Did you ever find a solution? I have the same problem. I’m a Premium user, so I have a support ticket open.

Note that my problem was caused by fapolicyd, as soon as I disabled that service, it was able to launch the gitaly server (or something) from /opt and the restore succeeded.

1 Like