Does Red Hat in-place OS upgrade corrupt GitLab CE?

I have a GitLab Community Edition v16.10.5 instance running on Red Hat Linux 7.9. I haven’t been able to migrate this instance using the prescribed GitLab backup and restore methods. Details on those woes can be found here.

I have started manually migrating users and groups to a new instance, and users will have to manually move their code. This will be an intensive process.

Has anyone had luck using Red Hat’s in-place upgrade process (leapp)? This would solve my issue until I’m forced to upgrade to RH9.

I use leapp to convert from centos 7 to oracle linux 8. I just stop gitlab, remove the gitlab package with rpm -e and after running oracle linux 8 install gitlab rpm and run gitlab-ctl reconfigure. I do this with gitlab ce 17.5.5. My database and redis runs out of gitlab vm and i convert too before with no problem.

When upgrading with leapp, you shouldn’t need to remove the Gitlab package with rpm -e, that’s gonna do a whole load of bad things, like remove the users from /etc/passwd and /etc/group and when Gitlab is reinstalled you can end up with a situation when UID and GID doesn’t match what was there previously. That means a load of problems with permissions and attempting to fix that under /var/opt/gitlab.

Using leapp will leave the Gitlab package installed because it’s a third-party repository not recognised by the leapp command. Leapp will give a warning about the package, but it won’t stop you running the upgrade. Only inhibitors will cause a blockage for leapp.

That said, you will want to disable the Gitlab repositories under /etc/yum.repos.d, so that it doesn’t attempt to use them and upgrade Gitlab to the latest version which would break your Gitlab installation. I’ve never personally tried a leapp upgrade on a Gitlab system, so make sure you do have a snapshot of your machine in case it goes wrong. Considering your experience attempting to migrate to a new server, this is going to be even more problematic if the leapp upgrade fails. You will then end up with a server that doesn’t work. If you don’t have a snapshot to revert the server from, you will not have a Gitlab installation at all.

On preleapp stage, leapp include gitlab to be removed (at least warned that will remove gitlab package). I moved gitlab repo files to a temporary folder before use leapp to avoid gitlab upgrade to latest gitlab version. Thats why i remove gitlab package. In my case uid or gid not change.

Yes, i have a lot of experience with leapp (many servers running, docker, mysql, postgresql, etc etc etc ), and of course i have snapshots/clones to rollback in case i need to repeat or change steps. This gitlab vm was the first with gitlab converted.

My gitlab is a big one and backup / restore need a lot of time to run.

My contribution to gitlab forum with this reply are to help people looking about leapp + gitlab.

Preleapp :

Removing dependent packages:
ansible noarch 2.9.27-1.el7 @System 103 M
gitlab-ce x86_64 17.5.5-ce.0.el7 @System 3.0 G
kernel-devel x86_64 3.10.0-1160.119.1.el7 @System 38 M
ovirt-guest-agent-common noarch 1.0.14-1.el7 @System 163 k
ruby-augeas x86_64 0.5.0-1.el7 @System 52 k
ruby-shadow x86_64 1:2.2.0-2.el7 @System 22 k

Gitlab install :

gitlab_gitlab-ce 495 B/s | 862 B 00:01
gitlab_gitlab-ce-source 384 B/s | 862 B 00:02
Dependencies resolved.

Package Arch Version Repository Size

Installing:
gitlab-ce x86_64 17.5.5-ce.0.el8 gitlab_gitlab-ce 1.2 G

Transaction Summary

Install 1 Package

Total size: 1.2 G
Installed size: 3.1 G
Downloading Packages:
[SKIPPED] gitlab-ce-17.5.5-ce.0.el8.x86_64.rpm: Already downloaded
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
Preparing : 1/1
Running scriptlet: gitlab-ce-17.5.5-ce.0.el8.x86_64 1/1
Installing : gitlab-ce-17.5.5-ce.0.el8.x86_64 1/1
Running scriptlet: gitlab-ce-17.5.5-ce.0.el8.x86_64 1/1
It looks like GitLab has not been configured yet; skipping the upgrade script.

   *.                  *.
  ***                 ***
 *****               *****
.******             *******
********            ********

,,
,
,
.,*******,
,
***,.
,,
.,
,
,
,.

 _______ __  __          __
/ ____(_) /_/ /   ____ _/ /_

/ / / / / / / __ `/ __
/ // / / // /
/ // / // /
_
//_/___/_,/./

Thank you for installing GitLab!
GitLab should be available at

For a comprehensive list of configuration options please see the Omnibus GitLab readme

Help us improve the installation experience, let us know how we did with a 1 minute survey:

Verifying : gitlab-ce-17.5.5-ce.0.el8.x86_64 1/1

Installed:
gitlab-ce-17.5.5-ce.0.el8.x86_64

Complete!

And good work too in doing that :slight_smile:

Although I also use leaap a lot, and removing Gitlab packages is usually destructive because the removal scripts then run, so I still have to warn about that so that people are aware. It may have worked for you, but it could work differently for others, hence the warning :slight_smile:

But yes, irrespective of the method, snapshots, backups are important. Also in taking into account his previous problems when repositories disappeared on restoring, he also needs to be extra careful, since as I mentioned he could end up with no Gitlab at all - especially since his backup methods haven’t worked so far. If the server is a VM, then he can make a snapshot easily enough. If it’s not a VM, then it can be even more problematic.

In which case, something like restic might be an idea to take a look into, of which I wrote about and documented here:

They could do this type of backup to an external USB drive using restic, and then if something does go wrong with the leapp upgrade, at least they can get back all the Gitlab data.

1 Like