Update Postgresql fails (13.14 -> 14.11)

Problem

I would like to update my Gitlab from version 16.11.10 to 17.3.4 and then to 17.4.1. Currently Postgresql 13.14 is used. To update Postgresql before the update, use the following command:
gitlab-ctl pg-upgrade

Checking for an omnibus managed postgresql: OK
Checking if postgresql['version'] is set: OK
Checking if we already upgraded: NOT OK
Checking for a newer version of PostgreSQL to install
Upgrading PostgreSQL to 14.11
Checking if disk for directory /var/opt/gitlab/postgresql/data has enough free space for PostgreSQL upgrade: OK
Checking if PostgreSQL bin files are symlinked to the expected location: OK
Toggling deploy page:cp /opt/gitlab/embedded/service/gitlab-rails/public/deploy.html /opt/gitlab/embedded/service/gitlab-rails/public/index.html
Toggling deploy page: OK
Toggling services:ok: down: alertmanager: 1s, normally up, want up
ok: down: crond: 0s, normally up
ok: down: gitaly: 0s, normally up
ok: down: gitlab-exporter: 0s, normally up
ok: down: gitlab-kas: 1s, normally up
ok: down: logrotate: 0s, normally up
ok: down: node-exporter: 1s, normally up
ok: down: postgres-exporter: 0s, normally up
ok: down: prometheus: 0s, normally up
ok: down: redis-exporter: 1s, normally up
ok: down: registry: 0s, normally up
ok: down: sidekiq: 0s, normally up
Toggling services: OK
Running stop on postgresql:ok: down: postgresql: 1s, normally up
Running stop on postgresql: OK
Symlink correct version of binaries: OK
Creating temporary data directory: OK
Initializing the new database: OK
Upgrading the data:Error upgrading the data to version 14.11
STDOUT: Performing Consistency Checks
-----------------------------
Checking cluster versions                                   ok
Checking database user is the install user                  ok
Checking database connection settings                       ok
Checking for prepared transactions                          ok
Checking for system-defined composite types in user tables  ok
Checking for reg* data types in user tables                 ok
Checking for contrib/isn with bigint-passing mismatch       ok
Checking for user-defined encoding conversions              ok
Checking for user-defined postfix operators                 ok
Checking for incompatible polymorphic functions             ok
Creating dump of global objects                             ok
Creating dump of database schemas
  gitlabhq_production

*failure*

Consult the last few lines of "pg_upgrade_dump_16386.log" for
the probable cause of the failure.
Failure, exiting
STDERR: 
Upgrading the data: NOT OK
== Fatal error ==
Error running pg_upgrade, please check logs
== Reverting ==
ok: down: postgresql: 5s, normally up
Symlink correct version of binaries: OK
ok: run: postgresql: (pid 3290288) 0s
== Reverted ==
== Reverted to 13.14. Please check output for what went wrong ==
Toggling deploy page:rm -f /opt/gitlab/embedded/service/gitlab-rails/public/index.html
Toggling deploy page: OK
Toggling services:ok: run: alertmanager: (pid 3290300) 0s
ok: run: crond: (pid 3290305) 0s
ok: run: gitaly: (pid 3290312) 1s
ok: run: gitlab-exporter: (pid 3290326) 0s
ok: run: gitlab-kas: (pid 3290332) 1s
ok: run: logrotate: (pid 3290339) 0s
ok: run: node-exporter: (pid 3290350) 0s
ok: run: postgres-exporter: (pid 3290355) 1s
ok: run: prometheus: (pid 3290361) 0s
ok: run: redis-exporter: (pid 3290372) 1s
ok: run: registry: (pid 3290379) 0s
ok: run: sidekiq: (pid 3290391) 1s
Toggling services: OK

The following error message appears in the file (pg_upgrade_dump_16386.log) (possibly more frequently in the same file):

command: "/opt/gitlab/embedded/postgresql/14/bin/pg_dump" --host /var/opt/gitlab/postgresql --port 50432 --username gitlab-psql --schema-only --quote-all-identifiers --binary-upgrade --format=custom  --file="pg_upgrade_dump_16386.custom" 'dbname=gitlabhq_production' >> "pg_upgrade_dump_16386.log" 2>&1
pg_dump: error: query returned 0 rows instead of one: SELECT typarray FROM pg_catalog.pg_type WHERE oid = '43240'::pg_catalog.oid;

How can I fix this error and update Postgresql?

Versions

  • gitlab-ce=16.11.10-ce.0
  • self-managed
  • postgresql 13.14

I had the same problem running gitlab in our kubernetes environment. Similar to a docker deployment, I mount the config, data and logs folders.
I don’t know why/how this fixed the problem, but here is what I did:
Upgrade path:
16.0.5 > 16.3.9 > 16.7.10 > 16.11.10 > 17.3.4 > 17.4.1

Perform upgrades until 16.7.10
Perform a gitlab-backup create

Stop the deployment or container.
mv data data.bk
mv logs logs.bk

mkdir data
mkdir logs

Start the deployment or container.
This should initialize a new clean deployment.
Copy your 16.7.10 backup file to data/backups/

Restore backup – you should see your postgresql verson is now 14.10
gitlab-ctl stop puma
gitlab-ctl stop sidekiq
gitlab-backup restore BACKUP= force=yes
Continue with normal upgrade path