Upgrading Gitlab CE from version 15.9.7 to 15.11.13 does not upgrade PostgreSQL from version 12.14 to 13.11

How to Use GitLab Upgrade
1

I recently upgraded Gitlab CE to 15.11.13 from 15.9.7 on two instances. The PostgreSQL upgrade happened for 1 instance from 12.12 to 13.11 but didn’t happen on the other. Both are not fresh installs.
There is this information given in the Database settings section of Gitlab Docs that the PostgreSQL will automatically upgrade to 13.11 unless the following conditions.

GitLab 15.11 and later

In GitLab 15.11, PostgreSQL will automatically be upgraded to 13.x except for the following cases:

The upgrade is skipped in any of the following cases:

  • You are running the database in high availability using Patroni.
  • Your database nodes are part of GitLab Geo configuration.
  • You have specifically opted out.
  • You have postgresql['version'] = 12 in your gitlab.rb

Fault-tolerant and Geo installations support manual upgrades to PostgreSQL 13, see Packaged PostgreSQL deployed in an HA/Geo Cluster.

None of the scenarios are applicable to my instances.

Could someone help me to find the reason as to why the PostgreSQL upgrade isn’t happening on one of the instance.

Thanks

2

Hey there,

Can you give a little bit more detail about your installation, e.g. installation type, OS, etc?

Also, if you could share some logs, would be useful.

It’s not mentioned here, but having GITLAB_SKIP_PG_UPGRADE=true as environment variable would also prevent it from updating.

There is another topic about this issue - please check if you’re having the same issue.

2 Likes
3

Hi @paula.kokic ,

I have a thread running here: Upgrading from 15.11.9 - #9 by developeremail3

I dont see any such env variable called GITLAB_SKIP_PG_UPGRADE being set. Even the version of PostgreSQL changed from version 12.12 to 12.14 from Gitlab 15.9.7 to 15.11.13.

My installation is Omnibus, Community Edition, OS would be CentOS.

Also I have two installations of similar configuration, even though for one instance the Postgres gets upgraded automatically while for the other it needs to be upgraded manually.

Also it would be great if you specify as to what kind of logs are you looking for?

Thanks

4

Okay. I believe GitLab service logs (from startup, as normally GitLab would / should attempt a Postgres update by itself) would be a good start to see if it even tries to update Postgres and if so, why does it not succeed or where does it get stuck.

5

Im not finding any logs in the postgresql logs

[root@dev-pat-sp-gitlab2 ~]# gitlab-ctl tail postgresql
==> /var/log/gitlab/postgresql/state <==

==> /var/log/gitlab/postgresql/current <==

Are there any way I can get the previous logs?

6

2023-08-25_06:33:03.75108 FATAL: terminating connection due to administrator command
2023-08-25_06:33:03.75203 LOG: background worker “logical replication launcher” (PID 8980) exited with exit code 1
2023-08-25_06:33:03.75211 FATAL: terminating connection due to administrator command
2023-08-25_06:33:03.75620 LOG: shutting down
2023-08-25_06:33:03.84377 LOG: database system is shut down
2023-08-25_06:33:17.93163 LOG: starting PostgreSQL 12.14 on x86_64-pc-linux-gnu, compiled by gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-44), 64-bit
2023-08-25_06:33:17.93453 LOG: listening on Unix socket “/var/opt/gitlab/postgresql/.s.PGSQL.5432”
2023-08-25_06:33:18.15257 LOG: database system was shut down at 2023-08-25 06:33:03 GMT
2023-08-25_06:33:18.16320 LOG: database system is ready to accept connections
2023-08-25_06:34:46.60576 LOG: no match in usermap “gitlab” for user “gitlab” authenticated as “root”
2023-08-25_06:34:46.60579 FATAL: Peer authentication failed for user “gitlab”
2023-08-25_06:34:46.60580 DETAIL: Connection matched pg_hba.conf line 70: “local all all peer map=gitlab”
2023-08-25_06:34:46.67659 LOG: no match in usermap “gitlab” for user “gitlab” authenticated as “root”
2023-08-25_06:34:46.67664 FATAL: Peer authentication failed for user “gitlab”
2023-08-25_06:34:46.67665 DETAIL: Connection matched pg_hba.conf line 70: “local all all peer map=gitlab”
2023-08-25_06:34:46.74409 LOG: no match in usermap “gitlab” for user “gitlab” authenticated as “root”
2023-08-25_06:34:46.74414 FATAL: Peer authentication failed for user “gitlab”
2023-08-25_06:34:46.74414 DETAIL: Connection matched pg_hba.conf line 70: “local all all peer map=gitlab”
2023-08-25_06:34:47.18261 received TERM from runit, sending INT instead to force quit connections
2023-08-25_06:34:47.18498 LOG: received fast shutdown request
2023-08-25_06:34:47.18774 LOG: aborting any active transactions
2023-08-25_06:34:47.19336 FATAL: terminating connection due to administrator command
2023-08-25_06:34:47.19338 FATAL: terminating connection due to administrator command

Doesn’t show much in the logs this time also it didnt upgrade PostgreSQL