I’m planning to migrate a Gitlab CE installation from CentOS 7 (EL7) server to Rocky Linux 8 (EL8) server. I’d rather not use gitlab-backup and restore, because I think that will take a considerable amount of time. I think I simply need to install the same (omnibus) version of Gitlab on the EL8 server, and then copy data from /var/opt/gitlab and /etc/gitlab to this new server. I did a test and that seems to have worked okay. My hope is I can just rsync the data in those directories when we’re ready to switch and everything will work as expected. Can anyone confirm that this is an acceptable way to migrate to a newer operating system version? The only potential concern I can think of is some binary data compatibility issue, but I don’t think that should be an issue since the omnibus packages should install the same versions of various Gitlab components on both EL7 and EL8. Thanks in advance for the help.
Yeah in theory this should work, curious, following.
Yes and no. The no part being that installing Gitlab will create users that might have different UID/GID than the original install. So rsync will not completely help in this instance, unless the users/groups in /etc/passwd and /etc/group have the same UID/GID as the original install.
Better is backup/restore, it’s not that difficult, and it’s just as quick.
Thanks for the feedback. From some brief reviewing of my test transfer, rsync -a seems to preserve the user/group permissions on files, even if the UIDs/GIDs are different between the two systems. This stack overflow post seems to confirm that rsync matches owners and groups by name by default.
The advantage of using rsync, as opposed to backup/restore, is that you can sync the data in advance and then simply do one last incremental sync during the outage window. That significantly reduces the outage window time required to migrate the data.
As I said, I appreciate the input and will perhaps try the backup and restore to see how long that would take.
Rsync does preserve permissions, but that UID/GID copied from the old system, may be associated to a completely different user. The users created during the installation of Gitlab may have different UID/GID than the original system. And this can be problematic.
I’ll give you an example, this is from my production server, looking at /etc/passwd:
gitlab-www:x:999:999::/var/opt/gitlab/nginx:/bin/false
git:x:998:998::/var/opt/gitlab:/bin/sh
gitlab-redis:x:997:997::/var/opt/gitlab/redis:/bin/false
gitlab-psql:x:996:996::/var/opt/gitlab/postgresql:/bin/sh
gitlab-prometheus:x:995:995::/var/opt/gitlab/prometheus:/bin/sh
now, let’s look at my test server:
gitlab-www:x:998:998::/var/opt/gitlab/nginx:/bin/false
git:x:997:997::/var/opt/gitlab:/bin/sh
gitlab-redis:x:996:996::/var/opt/gitlab/redis:/bin/false
gitlab-psql:x:995:995::/var/opt/gitlab/postgresql:/bin/sh
gitlab-prometheus:x:994:994::/var/opt/gitlab/prometheus:/bin/sh
you can clearly see that the UID’s do not match between the same usernames on the two servers. The big ones being git and gitlab-psql for a start. Therefore your rsync backup/restore possibly won’t work unless the UID/GID are exactly the same on both systems.
Some rsync parameters do suggest that it may preserve it, assuming of course the users already exist on the other side. I have attempted copying like you have in the past and it wasn’t successful and this was even with Gitlab fully stopped first and I also ensured that /opt/gitlab, /etc/gitlab, /var/opt/gitlab and /var/log/gitlab were copied. Some other systemd stuff most likely didn’t want to play nice, and I even created the users manually to ensure the UID/GID matched. Backup/restore worked quickly and effectively, but if you can get it to work for you, fair enough. I do understand the approach inasmuch that rsync allows you to copy again later much quicker for the only files that changed so I totally get it on why you want to do that.
Yeah, I understand the part about the UIDs/GIDs not matching, but I actually tested rsync between servers where the UIDs/GIDs didn’t match. After the sync, the file ownership (user and group) on the destination server looked correct. I also started the services aftewards and a very basic test (git clone/push, etc.) indicated that things were working as expected.
However, having said all that, I found this note in the backup documentation regarding system data transfers or snapshots:
Don’t use these methods to migrate from one operating system to another. The operating systems of the source and destination should be as similar as possible. For example, don’t use these methods to migrate from Ubuntu to Fedora.
Given that warning, I’m going to work towards trying to use the backup/restore function. Thanks again for the feedback.
At the end of last year we did a migration from one server to another (actually multiple, but that’s quite irrelevant).
We moved several things to object storage before.
We used rsync for repositories as it was A LOT faster than the backup+restore way that took all the repositories that are no longer used (for some reason, someone once though personal forks were the way to go, so every blocked user has a lot of those).
We used the documented backup+restore procedure (with SKIP=repositories added to the list of things that were skipped because they are on object storage) to get the DB transferred.
With many tests, that was quite possible, and we managed that switch in a couple of hours (backup+restore of the repositories would have taken 12 hours).
(We also moved from Ubuntu focal to Ubuntu jammy, so there was a new OS involved, but that was not the point)
Rsyncing the repositories and then backup+restoring everything else seems like a great compromise. Thanks so much for the suggestion.
I am trying to migrate from ubuntu 20.04 to 22.04.
The backup and restore instructions here Back up and restore GitLab | GitLab document both a backup+restore procedure, plus what appears to be a highly manual migration procedure. Can the I do the backup+restore, or do I have to do the migration?
I tried using rsync to copy /etc/gitlab and /var/opt/gitlab/git-data from 20.04 to 22.04, that did not work. On 22.04 gitlab started up, but it did not have any users or project repositories from 20.04 - it was still empty.
Those two directories don’t contain everything, e.g. the database is missing. A backup (for which you haven’t used SKIP to exclude anything) contains everything except for a couple of files in `/etc/gitlab/. The “highly manual migration procedure” is basically what you have to go through, and most of the steps listed are really quick.
I have a different scenario where my existing Gitlab is on RHEL 7 using LDAP. Now i need to move to RHEL 8 using ADS. How should the GID & UID combination work for gitlab backup and restore process.
You should probably have started a new topic instead of finding an old one and putting an unrelated question in that.
The backup+restore processes almost (just because I’m not sure) exclusively uses names, so as long as the correct users exists on the target it won’t matter if they have the same uid/gid.