Upgrading Community Edition from 14.4.1 to 14.5.0 hangs with postgresql FATAL error

Hi folks, I’ve hit an issue with the upgrade of the community edition from 14.4.1 to 14.5.0. The OS is 18.04.5 LTS and the command run was :

sudo apt-get install gitlab-ce=14.5.0-ce.0 -y

The upgrade hangs here :

bash[migrate gitlab-rails database] action run

In the postgresql logs :

2021-11-24_11:13:21.86345 DETAIL: Connection matched pg_hba.conf line 70: “local all all peer map=gitlab”
2021-11-24_11:13:21.91698 LOG: no match in usermap “gitlab” for user “gitlab” authenticated as “root”
2021-11-24_11:13:21.91699 FATAL: Peer authentication failed for user “gitlab”
2021-11-24_11:13:21.91699 DETAIL: Connection matched pg_hba.conf line 70: "local all all

Other snippet that might be relevant :

  • file[/opt/gitlab/service/postgresql/log/supervise/ok] action touch (skipped due to only_if)
    • file[/opt/gitlab/service/postgresql/supervise/status] action touch
      • change owner from ‘root’ to ‘gitlab-psql’
      • change group from ‘root’ to ‘gitlab-psql’
      • update utime on file /opt/gitlab/service/postgresql/supervise/status
    • file[/opt/gitlab/service/postgresql/log/supervise/status] action touch
      • change owner from ‘root’ to ‘gitlab-psql’
      • change group from ‘root’ to ‘gitlab-psql’
      • update utime on file /opt/gitlab/service/postgresql/log/supervise/status
    • file[/opt/gitlab/service/postgresql/supervise/control] action touch (skipped due to only_if)

We’ve done over a dozen upgrades on this server and this is the first time we’ve hit an issue. Any help or guidance is much appreciated. Thanks.

Hi,

sorry about the problems. The upgrading docs for 14.5.0 mention a background migration for MR diffs which might take longer.

Though, the error message with changing the user name looks like a bug or a missing config update for pg_hba.conf. Can you open a new bug report please?

Please also share which PostgreSQL version you using :slight_smile:

I’ve found a possible workaround when googling for the error message, maybe Gitlab-ce docker on apple m1 stuck on Recipe: gitlab::database_migrations - #13 by joe-huersch helps?

Cheers,
Michael

Thanks Michael. We are on version 12.7 of PostgreSQL. I’ll try and get a bug report created this weekend or Monday at the latest. I tried updating the user map as described in the workaround post but chef removes the entry and the error continues to show.

Bug logged here for reference : FATAL error message : Peer authentication failed for user "gitlab" when upgrading to 14.5.0 (#6539) · Issues · GitLab.org / omnibus-gitlab · GitLab

1 Like

Still digging into this :slight_smile: Tried the 14.5.1 bundle but still seeing the same behaviour.

Comparing production (14.4.1) with the failed staging (14.5.1) and the only difference I can find in file permissions is this :

During the 14.5.0 and 14.5.1 upgrade attempts the permissions on the “supervise/status” file are changed :

  • file[/opt/gitlab/service/postgresql/supervise/status] action touch
    - change owner from ‘root’ to ‘gitlab-psql’
    - change group from ‘root’ to ‘gitlab-psql’
    - update utime on file /opt/gitlab/service/postgresql/supervise/status

The permissions on staging look like this :

-rw-r--r-- 1 gitlab-psql gitlab-psql   20 Dec  2 12:23 status
-rw-r--r-- 1 root        root           4 Dec  2 12:22 stat
-rw-r--r-- 1 root        root           6 Dec  2 12:22 pid
prw------- 1 gitlab-psql gitlab-psql    0 Mar 13  2021 ok
-rw------- 1 root        root           0 Mar 13  2021 lock
prw------- 1 gitlab-psql gitlab-psql    0 Dec  2 12:23 control
drwxr-xr-x 6 root        root        4096 Dec  2 12:23 ..
drwxr-xr-x 2 root        root        4096 Dec  2 12:22 .

But on production the “supervise/status” file is still owned by root.

-rw-r--r-- 1 root        root          20 Nov  1 09:30 status
-rw-r--r-- 1 root        root           4 Nov  1 09:30 stat
-rw-r--r-- 1 root        root           5 Nov  1 09:30 pid
prw------- 1 gitlab-psql gitlab-psql    0 Oct 27  2017 ok
-rw------- 1 root        root           0 May 20  2016 lock
prw------- 1 gitlab-psql gitlab-psql    0 Nov  1 09:27 control
drwxr-xr-x 6 root        root        4096 Nov  1 09:22 ..
drwxr-xr-x 2 root        root        4096 Nov  1 09:30 .

I realise I’m clutching at straws at this point :slight_smile: If anyone else has seen this or found a workaround, please let me know. Thanks.

I have the same Issue, have you found solution for this?
thanks in advance,

1 Like

The solution/workaround is to allow the installer to timeout after 60 minutes, wait for the background migrations to complete and then rerun the installer.

apt-get install gitlab-ce=14.5.4-ce.0 -y
gitlab-rake db:migrate
apt-get install gitlab-ce=14.5.4-ce.0 -y

In my case the installer runs for 60 mins, the background migrations took around another 60 mins to completed and then the install completed quickly after that.

2 Likes

unfortunately it did not help either :frowning: