Upgrading Community Edition from 14.4.1 to 14.5.0 hangs with postgresql FATAL error

How to Use GitLab Upgrade
1

Hi folks, I’ve hit an issue with the upgrade of the community edition from 14.4.1 to 14.5.0. The OS is 18.04.5 LTS and the command run was :

sudo apt-get install gitlab-ce=14.5.0-ce.0 -y

The upgrade hangs here :

bash[migrate gitlab-rails database] action run

In the postgresql logs :

2021-11-24_11:13:21.86345 DETAIL: Connection matched pg_hba.conf line 70: “local all all peer map=gitlab”
2021-11-24_11:13:21.91698 LOG: no match in usermap “gitlab” for user “gitlab” authenticated as “root”
2021-11-24_11:13:21.91699 FATAL: Peer authentication failed for user “gitlab”
2021-11-24_11:13:21.91699 DETAIL: Connection matched pg_hba.conf line 70: "local all all

Other snippet that might be relevant :

  • file[/opt/gitlab/service/postgresql/log/supervise/ok] action touch (skipped due to only_if)
    • file[/opt/gitlab/service/postgresql/supervise/status] action touch
      • change owner from ‘root’ to ‘gitlab-psql’
      • change group from ‘root’ to ‘gitlab-psql’
      • update utime on file /opt/gitlab/service/postgresql/supervise/status
    • file[/opt/gitlab/service/postgresql/log/supervise/status] action touch
      • change owner from ‘root’ to ‘gitlab-psql’
      • change group from ‘root’ to ‘gitlab-psql’
      • update utime on file /opt/gitlab/service/postgresql/log/supervise/status
    • file[/opt/gitlab/service/postgresql/supervise/control] action touch (skipped due to only_if)

We’ve done over a dozen upgrades on this server and this is the first time we’ve hit an issue. Any help or guidance is much appreciated. Thanks.

2

Hi,

sorry about the problems. The upgrading docs for 14.5.0 mention a background migration for MR diffs which might take longer.

Though, the error message with changing the user name looks like a bug or a missing config update for pg_hba.conf. Can you open a new bug report please?

Please also share which PostgreSQL version you using :slight_smile:

I’ve found a possible workaround when googling for the error message, maybe Gitlab-ce docker on apple m1 stuck on Recipe: gitlab::database_migrations - #13 by joe-huersch helps?

Cheers,
Michael

3

Thanks Michael. We are on version 12.7 of PostgreSQL. I’ll try and get a bug report created this weekend or Monday at the latest. I tried updating the user map as described in the workaround post but chef removes the entry and the error continues to show.

4

Bug logged here for reference : FATAL error message : Peer authentication failed for user "gitlab" when upgrading to 14.5.0 (#6539) · Issues · GitLab.org / omnibus-gitlab · GitLab

1 Like
5

Still digging into this :slight_smile: Tried the 14.5.1 bundle but still seeing the same behaviour.

Comparing production (14.4.1) with the failed staging (14.5.1) and the only difference I can find in file permissions is this :

During the 14.5.0 and 14.5.1 upgrade attempts the permissions on the “supervise/status” file are changed :

  • file[/opt/gitlab/service/postgresql/supervise/status] action touch
    - change owner from ‘root’ to ‘gitlab-psql’
    - change group from ‘root’ to ‘gitlab-psql’
    - update utime on file /opt/gitlab/service/postgresql/supervise/status

The permissions on staging look like this :

-rw-r--r-- 1 gitlab-psql gitlab-psql   20 Dec  2 12:23 status
-rw-r--r-- 1 root        root           4 Dec  2 12:22 stat
-rw-r--r-- 1 root        root           6 Dec  2 12:22 pid
prw------- 1 gitlab-psql gitlab-psql    0 Mar 13  2021 ok
-rw------- 1 root        root           0 Mar 13  2021 lock
prw------- 1 gitlab-psql gitlab-psql    0 Dec  2 12:23 control
drwxr-xr-x 6 root        root        4096 Dec  2 12:23 ..
drwxr-xr-x 2 root        root        4096 Dec  2 12:22 .

But on production the “supervise/status” file is still owned by root.

-rw-r--r-- 1 root        root          20 Nov  1 09:30 status
-rw-r--r-- 1 root        root           4 Nov  1 09:30 stat
-rw-r--r-- 1 root        root           5 Nov  1 09:30 pid
prw------- 1 gitlab-psql gitlab-psql    0 Oct 27  2017 ok
-rw------- 1 root        root           0 May 20  2016 lock
prw------- 1 gitlab-psql gitlab-psql    0 Nov  1 09:27 control
drwxr-xr-x 6 root        root        4096 Nov  1 09:22 ..
drwxr-xr-x 2 root        root        4096 Nov  1 09:30 .

I realise I’m clutching at straws at this point :slight_smile: If anyone else has seen this or found a workaround, please let me know. Thanks.

6

I have the same Issue, have you found solution for this?
thanks in advance,

1 Like
7

The solution/workaround is to allow the installer to timeout after 60 minutes, wait for the background migrations to complete and then rerun the installer.

apt-get install gitlab-ce=14.5.4-ce.0 -y
gitlab-rake db:migrate
apt-get install gitlab-ce=14.5.4-ce.0 -y

In my case the installer runs for 60 mins, the background migrations took around another 60 mins to completed and then the install completed quickly after that.

2 Likes
8

unfortunately it did not help either :frowning: