Cannot upgrade on Centos 7.9

I have a gitlab instance on an old centos 7.9 which cannot be upgraded at this time as it would cause too much down time.

When trying to update, I get the following error which I cannot find a solution to by searching the net.
Wondering if someone might have a solution other than using any skip options.

Running transaction
gitlab preinstall: It seems you are upgrading from major version 11 to major version 14.
gitlab preinstall: It is required to upgrade to the latest 14.0.x version first before proceeding.
gitlab preinstall: Please follow the upgrade documentation at https://docs.gitlab.com/ee/update/index.html#upgrade                                              -paths
error: %pre(gitlab-ce-14.4.1-ce.0.el7.x86_64) scriptlet failed, exit status 1
Error in PREIN scriptlet in rpm package gitlab-ce-14.4.1-ce.0.el7.x86_64
  Verifying  : gitlab-ce-14.4.1-ce.0.el7.x86_64                                                               1/2
gitlab-ce-11.7.0-ce.0.el7.x86_64 was supposed to be removed but is not!
  Verifying  : gitlab-ce-11.7.0-ce.0.el7.x86_64                                                               2/2

Failed:
  gitlab-ce.x86_64 0:11.7.0-ce.0.el7                      gitlab-ce.x86_64 0:14.4.1-ce.0.el7

Yes, you can’t do that. You need to follow the upgrade path here: Upgrading GitLab | GitLab

Since you are on 11.7.0, then your path is:

11.11.8 → 12.0.12 → 12.1.17 → 12.10.14 → 13.0.14 → 13.1.11 → 13.8.8 → latest 13.12.Z → latest 14.0.Z → latest 14.Y.Z

So you need to start with 11.11.8 doing this:

yum install gitlab-ce-11.11.8-ce.0.el7.x86_64

you can use this link for finding the next version after you have upgraded to 11.11.8 and when you click that particular version it will give you in the top right hand corner, the yum install command similar to like I provided above: gitlab/gitlab-ce - Results for 'gitlab-ce' and el/7 in gitlab/gitlab-ce

also make sure background migrations have finished before attempting to upgrade to the next version on the list. This is important if you don’t want to break your install. This link direct to background migrations tells you how to check this from the console for example: Upgrading GitLab | GitLab

yum install gitlab-ce-11.11.8-ce.0.el7.x86_64

I started with the above which seemed to upgrade, no obvious errors but gitlab won’t come up now.
I’ve tried restarting the service and the server, nada.

Have to start digging I guess.

You can try running reconfigure

gitlab-ctl reconfigure
gitlab-ctl restart

Check also status so we can see if all processes started and also check the logs

gitlab-ctl status

I didn’t see any errors in any of the logs so I rebooted and that seemed to fix what ever the hickup was.
On to the next version… with another backup first :).

13.12.Z and 14.0.Z don’t seem to exist. Is it a typo?

No package gitlab-ce-13.12.Z-ce.0.el7.x86_64 available

That because you need to choose the last 13.12 release. Go to the package link I gave in my first post and search for 13.12. Then find the last release number and use that in place of Z.

I just didn’t get your example but all good now, running 14.1.6.
Thank you for the input on this, I appreciate it.

Is there a way to safely migrate to rocky linux 9 for example?

Gitlab only provides for Rocky 8 at the moment, but in reality you could install this on Rocky 9 - just means editing the repo files in /etc/yum.repos.d to change from 8 to 9 in the gitlab.repo file. Or just wait until Gitlab supports it by default. I used Debian 10 version on Debian 11 without problems until Gitlab natively supported it.

You would need to install the same version as on your old machine, and then use the Gitlab Backup and Restore documentation to take a backup and restore it on the new machine. You’ll also need to copy across the /etc/gitlab/gitlab.rb and /etc/gitlab/gitlab-secrets.json files from the old server to the new server.

Example, assuming you are still on CentOS 7 and using 14.1.6 from your post a year ago, then you upgrade this to the latest version which is 15.8.1. You will need to ensure to follow the Gitlab upgrade path to make sure important versions are not missed (direct upgrade from 14.1.6 to 15.8.1 will break your installation). Once upgraded, install Gitlab 15.8.1 on the new server (rocky 9), and on old machine make Gitlab backup as per docs, and then restore it on the new server. As mentioned, backup and restore can only be done to same version, so if backup was 14.1.6 it can only be restored to 14.1.6. So 15.8.1 backup is restored to 15.8.1 server.

It’s at 15.1.0 right now. That sounds like something I could try. Not much to lose since I would not be messing with the existing server other than using it’s backups.

I’m finally there after all this time. Looking at one tab 15.9.2 the old server and another tab with the new server of the same version.
The thing is, I don’t find a backup and restore option in admin yet read about that and looking online talks about using rsync.

Am I missing something?

The backups are done using console commands, see the docs: Back up and restore GitLab | GitLab

It’s better to read the official documentation than websites or whatever that talk about rsync or other rubbish that isn’t enough. You need a database backup as well, which rsync will not do.

It’s better to read the official documentation than websites or whatever that talk about rsync or other rubbish that isn’t enough. You need a database backup as well, which rsync will not do.

You mention having to backup the database but the doc says the backup takes care of that and that rsync is needed. I guess you’re saying that rsync is not enough even if it is used by the backup method.

Actually, that’s the doc I settled on but it’s a ton of information for someone that is not a gitlab expert. It’s all over the place for someone relatively new to gitlab.

There there’s
https://docs.gitlab.com/ee/raketasks/backup_gitlab.html
and the one you shared
https://docs.gitlab.com/ee/raketasks/backup_restore.html

Then just to add to the fun
https://docs.gitlab.com/ee/raketasks/restore_gitlab.html

I won’t bother pasting all the links but searching leads to so many articles, many of them from gitlab itself.

Is there any one doc that simply focuses on one way to backup and and restore for someone that doesn’t know everything just yet? :).

Yes, without seeing the articles you read before, you mentioned rsync, but no other steps. Using rsync to, eg: just copy /var/opt/gitlab would not be enough. The backup procedures in the document list the gitlab-backup create command. You linked the pages, which are sub-menus, so I don’t see what the problem is with the docs?

Segregating Backup and Restore, from the sub-pages of Backup, Restore is perfectly reasonable, rather than having one single long page that would be a mass of information.

But, in a nutshell, run the gitlab-backup create command, and then copy the archive from /var/opt/gitlab/backups to another system. Also remember /etc/gitlab/gitlab.rb and /etc/gitlab/gitlab-secrets.json since the backup doesn’t contain these.

Sorry, not trying to be a pain, we all have to start somewhere so I appreciate your help.
Just need a tiny bit more clarity which might help someone else finding this.

You mentioned backup up the database separately but don’t mention it above. I’m sure it’s because you’re used to it and know it well but for someone that’s not done it before, I’m not clear on the steps yet.

Use the tool, back it up, copy it to the new server.
Then? backup the database, restore that on the new server also or it would have been done by the tool?
Anything else, like ssl certs? For example, my old install doesn’t have letsencrypt but the one one does.

The gitlab-backup command includes everything unless you tell it to not include something - as mentioned in the docs, eg, not backing up container registry or other items. So it will have your database as well. What I wrote before was because you didn’t link what articles you read and just mentioned rsync. Rsync alone will not give you a backup.

SSL certificates if not managed by LetsEncrypt yes you have to backup yourself.

You use the gitlab backup command, then copy the backup from /var/opt/gitlab/backups to a different server, or download it to another machine or whatever. The same for gitlab.rb and gitlab-secrets.json in /etc/gitlab directory. That is all. SSL certificate you provided are outside the scope of the backup, so means you have to remember to copy and back them up somewhere. The Gitlab docs will not explain that, since it’s not Gitlab’s responsibility at this point. Their backup tool deals with the Gitlab side of things.

I really suggest reading the docs, it’s pretty easy to understand. Even examples are given and explained.

For sure, I’ll spend time reading. It’s just nice to have a real person’s input too.

Thanks for the information and clarity. I’ll get on this.

Generally, you only need to do these steps:

gitlab-backup create

an archive file will be created in /var/opt/gitlab/backups. You can then use scp/rsync or whatever other tools to your disposal to download the file from the server. You will also need to backup the files gitlab.rb and gitlab-secrets.json from /etc/gitlab directory. As seen here:

root@gitlab:~# ls /etc/gitlab/
config_backup  gitlab.rb  gitlab-secrets.json  ssl  trusted-certs

thos two gitlab files are important, else your backup will be useless. The config_backup, ssl and trusted-certs aren’t required unless your SSL certificates are stored here, then you will want to back them up as well. If you copied your certs to /etc/gitlab/ssl then include that in your backup.

There is one difference. The new server has a different dqdn.
I’ve edited the .rb file and don’t need to move the certs since they aren’t the same name.

Now on restoring, again, it’s not clear what I need to do in those docs. There are a LOT of ‘if this’ and ‘if that’ and massive amounts of options I don’t know that I need.

# ll
total 713060
-rw------- 1 git git 730173440 Mar  9 09:22 1678378917_2023_03_09_15.9.2_gitlab_backup.tar

$ sudo gitlab-backup restore BACKUP=1678378917_2023_03_09_15.9.2
2023-03-09 19:02:01 UTC -- Unpacking backup ...
2023-03-09 19:02:01 UTC -- Unpacking backup failed
2023-03-09 12:02:01 -0700 -- Deleting backup and restore lock file

$ sudo gitlab-backup restore BACKUP=1678378917_2023_03_09_15.9.2_gitlab_backup.tar
2023-03-09 17:31:31 UTC -- The backup file 1678378917_2023_03_09_15.9.2_gitlab_backup.tar_gitlab_backup.tar does not exist!
2023-03-09 10:31:31 -0700 -- Deleting backup and restore lock file

$ sudo gitlab-backup restore
2023-03-09 18:42:05 UTC -- Unpacking backup ...
2023-03-09 18:42:05 UTC -- Unpacking backup failed
2023-03-09 11:42:05 -0700 -- Deleting backup and restore lock file

I also tried the full path with no difference. In all those backup/restore articles, it’s not clear how to get this done without wrecking something. I found some other articles that seem to imply this is all I need to do but since it’s not working, I don’t know.

The log shows;
{“severity”:“INFO”,“time”:“2023-03-09T20:09:01.267Z”,“correlation_id”:null,“message”:“The backup file 1678378917_2023_03_09_15.9.2_gitlab_backup.tar_gitlab_backup.tar does not exist!”}

I found this;

I don’t think my version uses ‘rake’ but figured I had nothing to lose;
$ gitlab-rake gitlab:backup:restore
2023-03-09 20:53:51 UTC – Unpacking backup …
2023-03-09 20:53:51 UTC – Unpacking backup failed
2023-03-09 13:53:51 -0700 – Deleting backup and restore lock file

The tar file is fine. I checked it.

I installed using yum/dnf;

I know I have community version but haven’t seen anything on the server telling me if it’s Omnibus or something else.