Pg_dump: [archiver (db)] connection to database "gitlabhq_production" failed: could not connect to server: No such file or directory

How to Use GitLab Upgrade
1

Problem to solve

Upgraded to gitlab-ee:16.11.8-ee.0 and intermittently I am getting the follow error

pg_dump: [archiver (db)] connection to database "gitlabhq_production" failed: could not connect to server: No such file or directory
	Is the server running locally and accepting
	connections on Unix domain socket "/var/opt/gitlab/postgresql/.s.PGSQL.5432"?
rake aborted!
Backup::Error: Backup failed
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/database.rb:50:in `dump'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/backup.rake:99:in `block (4 levels) in <top (required)>'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/backup.rake:10:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Tasks: TOP => gitlab:backup:db:create
(See full trace by running task with --trace)

Steps to reproduce

I always walk the upgrade release, 16.11.0-ee.0, 16.11.1-ee.0, 16.11.2-ee.0, …, 16.11.8-ee.0 using this scripted edited for each release

#!/usr/bin/env bash

export GITLAB_HOME=/srv/gitlab

docker container stop gitlab
if [ $? -ne 0 ]; then
    echo "Failed to stop gitlab container. Aborting."
    exit 1
fi

docker container rm --force gitlab
if [ $? -ne 0 ]; then
    echo "Failed to remove the gitlab container. Aborting."
    exit 1
fi

docker run --detach \
  --hostname gitlab.x.x.com \
  --publish 443:443 --publish 80:80 --publish 22:22 --publish 5005:5005 \
  --name gitlab \
  --restart always \
  --volume $GITLAB_HOME/config:/etc/gitlab \
  --volume $GITLAB_HOME/logs:/var/log/gitlab \
  --volume $GITLAB_HOME/data:/var/opt/gitlab \
  --shm-size 256m \
  --log-opt max-size=1024m \
  gitlab/gitlab-ee:16.11.8-ee.0

if [ $? -ne 0 ]; then
    echo "Failed to start container. Aborting."
    exit 1
 fi

docker logs -f gitlab

I’ve tried this command for more verbose output, but I suspect the CRON=1 squelches verbose output

$ sudo /opt/gitlab/bin/gitlab-rake gitlab:backup:create  --verbose CRON=1

Configuration

Provide screenshots from the GitLab UI showing relevant configuration, if applicable.
On self-managed instances, add the relevant configuration settings or changes.

Versions

Versions

% sudo gitlab-rake gitlab:env:info

System information
System:		
Proxy:		no
Current User:	git
Using RVM:	no
Ruby Version:	3.1.4p223
Gem Version:	3.5.7
Bundler Version:2.5.8
Rake Version:	13.0.6
Redis Version:	7.0.15
Sidekiq Version:7.1.6
Go Version:	unknown

GitLab information
Version:	16.11.8-ee
Revision:	b4da02ea0de
Directory:	/opt/gitlab/embedded/service/gitlab-rails
DB Adapter:	PostgreSQL
DB Version:	14.11
URL:		https://gitlab.x.x.com
HTTP Clone URL:	https://gitlab.x.x.com/some-group/some-project.git
SSH Clone URL:	git@gitlab.x.x.com:some-group/some-project.git
Elasticsearch:	no
Geo:		no
Using LDAP:	no
Using Omniauth:	yes
Omniauth Providers:

GitLab Shell
Version:	14.35.0
Repository storages:
- default: 	unix:/var/opt/gitlab/gitaly/gitaly.socket
GitLab Shell path:		/opt/gitlab/embedded/service/gitlab-shell

Gitaly
- default Address: 	unix:/var/opt/gitlab/gitaly/gitaly.socket
- default Version: 	16.11.8
- default Git Version: 	2.43.5

No issues found on gitlab:check (snippet for brevity)

gitlab-rake gitlab:check
Checking GitLab subtasks ...

Checking GitLab Shell ...

GitLab Shell: ... GitLab Shell version >= 14.35.0 ? ... OK (14.35.0)
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Internal API available: OK
Redis available via internal API: OK
gitlab-shell self-check successful

Checking GitLab Shell ... Finished
<snip>

All Status are up (snippet for brevity)

sudo gitlab-rake db:migrate:status

database: gitlabhq_production

 Status   Migration ID    Migration Name
--------------------------------------------------
   up     20121220064453  ********** NO FILE **********
   up     20130102143055  ********** NO FILE **********