Problem to solve
While upgrading the embedded PostgreSQL instance to v13, we receive an error:
command: "/opt/gitlab/embedded/postgresql/13/bin/pg_dump" --host /var/opt/gitlab/postgresql --port 50432 --username gitlab-psql --schema-only --quote-all-identifiers --binary-upgrade --format=custom --file="pg_upgrade_dump_16386.custom" 'dbname=gitlabhq_production' >> "pg_upgrade_dump_16386.log" 2>&1
pg_dump: error: query returned 0 rows instead of one: SELECT typarray FROM pg_catalog.pg_type WHERE oid = '68945'::pg_catalog.oid;
Steps to reproduce
We see the same issue appearing in two scenarios. They are both ending on the same PosgreSQL upgrade error:
- Run upgrade GitLab from 15.4.6 to 15.11.13 according to the recommended upgrade path (with apt install)
- Or, after manually upgrading the instance step-by-step we managed to get to 15.11.13, next GitLab version (16.3.8) requires Postgresql 13, so we need to run: gitlab-ctl pg-upgrade which fails with the abovementioned error.
What we’ve tried so far
- First, we tried to do the recommended upgrade from v. 15.4.6 to 15.11.13
- Then did a step-by-step upgrade, that allowed us to reach our current version 15.11.13 (without any issues)
- Tried to manually upgrade PostgreSQL with gitlab-ctl pg-upgrade (also tried with “-V 13” without any luck):
# gitlab-ctl pg-upgrade
Checking for an omnibus managed postgresql: OK
Checking if postgresql['version'] is set: OK
Checking if we already upgraded: NOT OK
Checking for a newer version of PostgreSQL to install
Upgrading PostgreSQL to 13.11
Checking if disk for directory /var/opt/gitlab/postgresql/data has enough free space for PostgreSQL upgrade: OK
Checking if PostgreSQL bin files are symlinked to the expected location: OK
Waiting 30 seconds to ensure tasks complete before PostgreSQL upgrade.
See https://docs.gitlab.com/omnibus/settings/database.html#upgrade-packaged-postgresql-server for details
If you do not want to upgrade the PostgreSQL server at this time, enter Ctrl-C and see the documentation for details
Please hit Ctrl-C now if you want to cancel the operation.
Toggling deploy page:cp /opt/gitlab/embedded/service/gitlab-rails/public/deploy.html /opt/gitlab/embedded/service/gitlab-rails/public/index.html
Toggling deploy page: OK
Toggling services:ok: down: alertmanager: 1s, normally up
ok: down: gitaly: 0s, normally up
ok: down: gitlab-exporter: 0s, normally up
ok: down: gitlab-kas: 0s, normally up
ok: down: grafana: 0s, normally up
ok: down: logrotate: 0s, normally up
ok: down: node-exporter: 1s, normally up
ok: down: postgres-exporter: 0s, normally up
ok: down: prometheus: 1s, normally up
ok: down: redis-exporter: 0s, normally up
ok: down: sidekiq: 0s, normally up
Toggling services: OK
Running stop on postgresql:ok: down: postgresql: 0s, normally up
Running stop on postgresql: OK
Symlink correct version of binaries: OK
Creating temporary data directory: OK
Initializing the new database: OK
Upgrading the data:Error upgrading the data to version 13.11
STDOUT: Performing Consistency Checks
-----------------------------
Checking cluster versions ok
Checking database user is the install user ok
Checking database connection settings ok
Checking for prepared transactions ok
Checking for system-defined composite types in user tables ok
Checking for reg* data types in user tables ok
Checking for contrib/isn with bigint-passing mismatch ok
Creating dump of global objects ok
Creating dump of database schemas
gitlabhq_production
*failure*
Consult the last few lines of "pg_upgrade_dump_16386.log" for
the probable cause of the failure.
Failure, exiting
STDERR:
Upgrading the data: NOT OK
== Fatal error ==
Error running pg_upgrade, please check logs
== Reverting ==
ok: down: postgresql: 16s, normally up
Symlink correct version of binaries: OK
ok: run: postgresql: (pid 605258) 1s
== Reverted ==
== Reverted to 12.14. Please check output for what went wrong ==
Toggling deploy page:rm -f /opt/gitlab/embedded/service/gitlab-rails/public/index.html
Toggling deploy page: OK
Toggling services:ok: run: alertmanager: (pid 605274) 0s
ok: run: gitaly: (pid 605291) 1s
ok: run: gitlab-exporter: (pid 605312) 0s
ok: run: gitlab-kas: (pid 605315) 1s
ok: run: grafana: (pid 605328) 0s
ok: run: logrotate: (pid 605342) 1s
ok: run: node-exporter: (pid 605348) 0s
ok: run: postgres-exporter: (pid 605354) 0s
ok: run: prometheus: (pid 605363) 1s
ok: run: redis-exporter: (pid 605377) 0s
ok: run: sidekiq: (pid 605386) 1s
Toggling services: OK
- Run the query manually to validate the issue:
# sudo gitlab-psql -c "SELECT typarray FROM pg_catalog.pg_type WHERE oid = '68945'::pg_catalog.oid;"
typarray
----------
(0 rows)
- I checked with other instances of GitLab and PostgreSQL to see how this row should look, but I found no traces of such an oid on any of our embedded instances or standalones.
- Did the “gitlab-ctl reconfigure” command a few times.
- Obviously, tried to look for any kind of information on this particular datatype in PostgreSQL or an error it throws somewhere on the net and on forums. It seems to be quite a peculiar problem.
Versions
Please select whether options apply, and add the version information.
- Self-managed, Enterprise Edition
- GitLab.com SaaS
Versions:
System information
System: Ubuntu 20.04
Proxy: no
Current User: git
Using RVM: no
Ruby Version: 3.0.6p216
Gem Version: 3.2.33
Bundler Version:2.3.15
Rake Version: 13.0.6
Redis Version: 6.2.11
Sidekiq Version:6.5.7
Go Version: unknown
GitLab information
Version: 15.11.13-ee
Revision: c2a3ebb864f
Directory: /opt/gitlab/embedded/service/gitlab-rails
DB Adapter: PostgreSQL
DB Version: 12.14
URL: https://lab-git.-----.it
HTTP Clone URL: https://lab-git.-----.it/some-group/some-project.git
SSH Clone URL: git@lab-git.-----.it:some-group/some-project.git
Elasticsearch: no
Geo: no
Using LDAP: yes
Using Omniauth: yes
Omniauth Providers:
GitLab Shell
Version: 14.18.0
Repository storages:
- default: unix:/var/opt/gitlab/gitaly/gitaly.socket
GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell