What's the best way to upgrade Gitlab to the latest release and change OS?

Hello,

As a Linux administrator first and foremost, and knowing little about the Git/Gitlab environment, I need to migrate our Gitlab installation to a new Linux platform, in this case from CentOS to Debian…
The current server hasn’t benefited from regular OS upgrades, and even less from regular upgrades for its Gitlab part…
I’m not sure what’s the best way to proceed!

current Linux version: CentOS Linux release 7.3.1611 (Core)
current Gitlab version: GitLab Community Edition 11.0.1

desired versions after migration
Linux version: Debian 12
Gitlab version: latest available

1st option considered:

1. yum update on the current linux version.
   yum update returns the following available version: gitlab-ce x86_64 16.11.2-ce.0.el7 gitlab_official

Which would suit me perfectly!

That said, given the obsolescence of the server and the Gitlab version currently in use, the yum command returns a substantial number and size of packages to be updated:

Transaction Summary
=============================================================================Install 12 Packages (+52 Dependent packages)
Upgrade 409 Packages
Remove 1 Package

Total download size: 1.3 G

Having read a lot of documentation about failed Gitlab upgrades, do you think I can jump from Gitlab version 11.0.1 to version 16.11 without problems ?
The Gitlab documentation in any case recommends a gradual upgrade, but not a jump from several versions in a single upgrade!
What’s more, given that the Linux distribution currently in use (CentOS) is obsolete and being discontinued by its vendor, a package upgrade seems rather risky to me… (there are errors during the “yum update” attempt…)

2nd option considered:
full backup (using the tools provided by Gitlab) of the current configuration, according to available documentation, for example:

1. installation of the new Debian server version 12
2. installation of the Gitlab version identical to the one installed on the old server (GitLab Community Edition 11.0.1)
3. restore the Gitlab backup previously performed on the CentOS server
4. then progressive upgrade from Gitlab 11.x to 16.x

Potential problems:

1. there will most likely be compatibility problems installing Gitlab version 11 on a Debian 12 server
2. Will the Gitlab backup performed on CentOS be fully compatible with a Gitlab installation on a Debian server ?

Are the 2 steps proposed above coherent and potentially feasible?

thank you in advance for your patience in reading this far… and I wish you an excellent day !

Jean

There is no package for Gitlab 11 on Debian 12. So you will not be able to do that. The earliest Debian 12 package is Gitlab 16.1.

You will have to upgrade Gitlab on your EL7 box until you can get to the latest possible version - you mention 16.11.2. Once you are at 16.11.2 on EL7, then install Gitlab 16.11.2 on Debian 12. Copy /etc/gitlab/gitlab.rb and /etc/gitlab/gitlab-secrets.json from the EL7 server to the Debian 12 server. Run gitlab-ctl reconfigure to get an empty installation of Gitlab running. Now you can do the backup/restore as the Gitlab documentation.

Hello iwalker,

my current installation is 11.0.1, I didn’t find this release in the upgrade tool, the closest being 11.06

Google search “download gitlab-ce 11 package for centos 7” returns the following pages:

etc…
etc…
etc…

Then I’ll be able to take advantage of the tool: Upgrade Path
and all its documentation.

Many thanks for your help!

Jean

You can use the old documentation to find the upgrade path: Upgrading GitLab | GitLab

as you can see here, you are on a version between 10.8.7 and 11.11.8. Therefore from your current version, your next upgrade is 11.11.8. I suggest taking a backup or making a VM snapshot of your server before doing the upgrade just in case problems are encountered.

Oh, I hadn’t seen that…

indeed, it will save the number of upgrades needed…
And of course, yes, a good snapshot before each subsequent upgrade…

thanks again!

Jean

Just for information, I’ll give you feedback in 3 weeks… in the meantime I’m going on vacation…

Hello, I leave you the update sequence to reach the gitlab-ce-16.11.2-ce.0.el7 version
You can see the list of available versions with the command yum --showduplicates list gitlab-ce

yum upgrade gitlab-ce-8.6.2-ce.0.el7 -y && sleep 300

yum upgrade gitlab-ce-8.7.0-ce.0.el7 -y && sleep 300

yum upgrade gitlab-ce-8.8.0-ce.0.el7 -y && sleep 300

yum upgrade gitlab-ce-8.9.0-ce.0.el7 -y && sleep 300

yum upgrade gitlab-ce-8.10.0-ce.1.el7 -y && sleep 300

yum upgrade gitlab-ce-8.11.0-ce.0.el7 -y && sleep 300

yum upgrade gitlab-ce-8.12.0-ce.0.el7 -y && sleep 300

yum upgrade gitlab-ce-8.17.8-ce.0.el7 -y && sleep 300

yum upgrade gitlab-ce-9.0.0-ce.0.el7 -y && sleep 300

yum upgrade gitlab-ce-9.5.10-ce.0.el7 -y && sleep 300

yum upgrade gitlab-ce-10.0.0-ce.0.el7 -y && sleep 300

yum upgrade gitlab-ce-10.8.7-ce.0.el7 -y && sleep 300

yum upgrade gitlab-ce-11.0.0-ce.0.el7 -y && sleep 300

yum upgrade gitlab-ce-11.11.8-ce.0.el7 -y && sleep 300

yum upgrade gitlab-ce-12.0.0-ce.0.el7 -y && sleep 300

yum upgrade gitlab-ce-12.10.14-ce.0.el7 -y && sleep 300

yum upgrade gitlab-ce-13.0.0-ce.0.el7 -y && sleep 300

yum upgrade gitlab-ce-13.12.15-ce.0.el7 -y && sleep 300

yum upgrade gitlab-ce-14.0.0-ce.0.el7 -y && sleep 300

yum upgrade gitlab-ce-14.0.12 -y && sleep 300

yum upgrade gitlab-ce-14.1.1-ce.0.el7 -y && sleep 300

yum upgrade gitlab-ce-14.1.8-ce.0.el7 -y && sleep 300

yum upgrade gitlab-ce-14.2.1-ce.0.el7 -y && sleep 300

yum upgrade gitlab-ce-14.2.7-ce.0.el7 -y && sleep 300

yum upgrade gitlab-ce-14.3.0-ce.0.el7 -y && sleep 300

yum upgrade gitlab-ce-14.3.6 -y && sleep 300

yum upgrade gitlab-ce-14.9.5 -y && sleep 300

yum upgrade gitlab-ce-14.10.5 -y && sleep 300

yum upgrade gitlab-ce-15.0.5 -y && sleep 300

yum upgrade gitlab-ce-15.4.6 -y && sleep 300

yum upgrade gitlab-ce-15.11.13 -y && sleep 300

yum upgrade gitlab-ce-16.1.6 -y && sleep 300

yum upgrade gitlab-ce-16.3.7 -y && sleep 300

yum upgrade gitlab-ce-16.7.7 -y && sleep 300

yum upgrade gitlab-ce-16.10.0 -y && sleep 300

yum upgrade gitlab-ce-16.11.1-ce.0.el7 -y && sleep 300

yum upgrade gitlab-ce-16.11.2-ce.0.el7 -y

Using sleep is a very bad idea. Some database migrations take a lot longer than this. By using that method you have the potential of breaking your installation before the migrations have finished. In fact, people who have ran the next upgrade before migrations finished broke their install. Then you will be looking at restoring your system from a backup.

hello msmartinez,

interesting…
but, total upgrade, in a few lines?
Without the usual checks, between each version change ?
All this scares me a bit…

thanks for the tip, but being cautious by nature, I think I’ll carry out the operation in stages, with checks between each upgrade…

best regards

Jean

Hello everyone,

Given my limited knowledge of the product, I still have a number of unanswered questions…

GITLAB SERVER
current Postgresql version on Gitlab server:
PostgreSQL 9.6.8 on x86_64-pc-linux-gnu, compiled by gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-16), 64-bit

Since the schema normally used by the Gitlab software (bd gitlabhq_production) is almost empty of data, is this LOCAL DB still used by Gitlab, perhaps for internal management purposes?

If it’s not being used, can it be stopped? (As mentioned below, the PRODUCTION DB is located on a remote server).

Given that this LOCAL Postgres DB will be automatically upgraded at the same time as the Gitlab version, should I also upgrade the Postgresql version on the remote machine?

Does upgrading the Gitlab software change anything in the Postgres schema on the remote DB? (e.g. tables, views or stored procedures)?

If an upgrade of the Postgresql SOFTWARE on the REMOTE machine is required, can I do this AFTER the Gitlab upgrades have been completed?

My current configuration for Gitlab access to the REMOTE Postgresql database:
(configuration file: /etc/gitlab/gitlab.rb)
→

############################
# gitlab.yml configuration #
############################
gitlab_rails['db_adapter'] = "postgresql"
gitlab_rails['db_database'] = "gitlab_db"
gitlab_rails['db_host'] = "srv-postgres-02.dsr.ch"

POSTGRESQL SERVER (srv-postgres-02.dsr.ch)
Postgresql version on DISTANT Postgresql server:
PostgreSQL 9.5.5 on x86_64-pc-linux-gnu, compiled by gcc (GCC) 4.4.7 20120313 (Red Hat 4.4.7-17), 64-bit

After verification on a test Gitlab installation on Debian 12, a recent Gitlab release uses version 14.x of Postgress…
But would a recent Gitlab release run with PostgreSQL 9.5.5?

thanks again for your help and have a nice day

Jean

No it won’t. Postgresql needs to be updated. The Gitlab documentation does list what Postgres versions are supported by each Gitlab version.

oops…
since the remote DB server hosts many schemas other than the gitlab schema, I’ll have to isolate the gitlab schema on a dedicated server and upgrade the remote gitlab schema separately, at the same time as the Gitlab software itself…
or clone the current postgres server, change its IP and hostname before upgrading Postgres software…

lots of complications, I’ll evaluate the best solution!

thanks for the info

Jean

It’s probably not impossible to get this migration done by installing an old release of Debian that has GitLab 11 (I don’t know which); move to that, upgrade GitLab, upgrade Debian, … but while it does give more experience with Debian, some of that experience might be specific to those version upgrades (also depends on your hardware), so I’m not recommending that over what @iwalker says: Upgrade GitLab on your Centos box, move to Debian and upgrade GitLab further there.
And 11.0.6 (try to remember the points, it doesn’t matter in this case but I have seen a case where it did) is just a few point releases from 11.0.1 and I believe no potentially breaking changes happen in point releases, so what ever the upgrade tool says for 11.0.6 is most likely also relevant for you 11.0.1. But @iwalker has already dug up the old (i.e. before the tool was made) upgrade path.

hello everyone,

@Grove, thank you for your idea of porting everything to an older Debian version first.
The approach suggested by @iwalker, of upgrading Gitlab to CentOS and then migrating to Debian also sounds like a good idea to me.

Honestly, I don’t know yet what approach I’m going to take…
Before that, I’ve got a double challenge:
updating the Gitlab server on the one hand, and on the other
update the DISTANT Postgresql server, otherwise Gitlab won’t work.

My BIG problem is that the remote server hosts several other critical application databases, which of course I can’t update at the same time… (that’s what happens when you must deal with a computer park that’s never been seriously maintained… )

Knowing that the Gitlab database is currently hosted on a remote Postgres server, would it be possible for me to relocate the remote Gitlab DB, on the Gitlab server itself?

If I check my current configuration:

############################
# gitlab.yml configuration #
############################
gitlab_rails['db_adapter'] = "postgresql"
gitlab_rails['db_database'] = "gitlab_db"
gitlab_rails['db_host'] = "srv-postgres-02.dsr.ch"

The current contents of the "gitlab_db" schema on my Gitlab server, it is empty of any content:

[root@srv-gitlab-01 ~]# gitlab-psql
psql (9.6.8)
Type "help" for help.
gitlab_db=# \d
No relations found.
gitlab_db=#

The content of the "gitlab_db" schema on the REMOTE server: “srv-postgres-02.dsr.ch”, → there is the PRODUCTION data

gitlab_db=# \connect gitlab_db;
psql (8.4.20, server 9.5.5)
WARNING: psql version 8.4, server version 9.5.
         Some psql features might not work.
You are now connected to database "gitlab_db".
gitlab_db=#

My idea:
From remote server, “srv-postgres-02.dsr.ch”: pg_dump gitlab_db

From Gitlab server, srv-gitlab-01: pg_restore gitlab_db (of previously backed-up schema)
AND Configuration file corrected to point to the now LOCAL bd
gitlab_rails['db_host'] = "srv-gitlab-01" OR (“localhost”)

Since a Gitlab upgrade automatically updates the Postgres release supplied with the Gitlab package, this would simplify the operation enormously…

Is it possible to do this?
Is it a good idea to relocate the DB to the Gitlab server itself?
If so, is the gitlab_rails['db_host'] = "localhost"
parameter the only one that needs to be changed to modify access to the DB?

Thanks a log again !

Jean

I’m not sure you will be able to do that, but I could be wrong. You can dump the database from the remote postgres server. However, you will need to reconfigure Gitlab before you attempt to restore it on the Gitlab server. You will need to ensure postgres is configured to use the local on which is currently empty. Once gitlab.rb has been amended, you can then run gitlab-reconfigure followed by gitlab-ctl restart postgresql to ensure that has been restarted.

Then at this point you would use the /opt/gitlab/embedded/bin/pg_restore. But again, I’m not sure how Gitlab will behave after the reconfigure with an empty database until it’s been restored.

It would probably be easier to do what you mentioned before. Clone the VM with the postgresql 9.x version, and start the cloned VM on a different IP address. You could remove all other db’s from this server other than the gitlab one. Then just reconfigure Gitlab to use this one instead of the original remote one. You may need to check/reconfigure postgres to listen on a different IP once the VM has been cloned unless it was using listen = *

Either way, you can try both and attempt to bring the DB back to the local postgres on Gitlab but I’m not sure how successful this will be. Try and see what happens, or clone the VM and do it that way.

In fact, it’s looking more and more like a nightmare…

I had forgotten the fact that the current database server is running CentOS 6.6 !!!
In other words, a version that’s been obsolete since the dawn of time…
Needless to say, it will be extremely difficult to update the Postgresql 9.5 version currently running on this server.

So… here’s the current situation:
Gitlab server: CentOS version 7.3 → end of life in June 2024, so very soon!
Postgresql server for DB Gitlab: version 6.6 → end of life a long time ago.

Under these conditions, it seems complicated to test my idea of cloning the Postgres server and evolving its Postgresql versions via successive upgrades, as access to obsolete packages is somewhat random, as is the management of dependencies relating to the installation of the postgres RPM package.

Although I haven’t compiled any sources for a long time, I’m considering the following:
PostgreSQL: File Browser

In other words:

1. Install a Debian server with the latest release (12)
2. Compile the Postgresql version identical to the one currently in use, i.e. 9.5
3. pg_backup the Gitlab database
4. pg_restore Gitlab DB on new Debian server
5. Reconfigure Gitlab connection–>Postgresql (gitlab_rails[‘db_host’] = “New Debian server”)
6. Functional tests after moving the DB to the new server
7. Then, successive Gitlab and Postgresql upgrades.

Unlike precompiled .deb or .rpm packages, which are difficult to mix between different versions of the same OS, it seems to me that I can recompile Postgresql versions, even older ones, on a recent server if I’m not mistaken?

Probably easier is to install a new server with Rocky 8 for example, since from the Postgres community repositories you have access to 9.6, 10, 12, 13, 14 and 15. (community repos: PostgreSQL: Linux downloads (Red Hat family))

Install Postgresql 9.6 and pg_restore your Gitlab database from the 9.5 server. Change the Gitlab config to point to the new server with Postgresql 9.6. Upgrade your Gitlab as far as you can go before you need to change the Postgres version again. Then at this point go to 10 or 12 depending on what version is required. There are instructions to migrate the data from 9.6 to 10/12 without having to pg_dump beforehand which can be found easily enough with Google or whatever. Then after this, start the next upgrade on the path and go as far as possible before you need to upgrade to 13, 14 or 15.

Another alternative with Rocky 8 is that the modules exist so you don’t need to use the postgres community repositories, so:

[root@rocky8 ~]# dnf module list postgresql
Last metadata expiration check: 3:01:54 ago on Tue 28 May 2024 01:03:50 PM CEST.
Rocky Linux 8 - AppStream
Name                      Stream               Profiles                         Summary                                          
postgresql                9.6                  client, server [d]               PostgreSQL server and client module              
postgresql                10 [d]               client, server [d]               PostgreSQL server and client module              
postgresql                12                   client, server [d]               PostgreSQL server and client module              
postgresql                13                   client, server [d]               PostgreSQL server and client module              
postgresql                15                   client, server [d]               PostgreSQL server and client module              

there are instructions for how to do upgrades when switching between the modules. These can also be found easily enough with a quick google. If you need PostgreSQL 14 at some point, then you will need to use the PostgreSQL community repositories since 14 doesn’t exist in the modules available with EL distros by default. My Gitlab is on the latest, and it’s using 14. So whilst you can use the modules that are available with dnf by default it may actually be better just installing and using the PostgreSQL repositories which will allow you to choose between all the versions you will ever need.

Rocky 8 will go end of life in 2029, so you’ll be OK for about 5 years.

Great idea!

Not knowing Rocky Linux at all, I didn’t even think about this option…

Many thanks again for your help!

Yes, I left it as an example, but depending on the type of installation and number of users you could adjust the parameter, that upgrade set worked for me.

Greetings.