Postgres upgrade from 9.6.11 to 10.9 fails (omnibus-gitlab)


When upgrading from 12.1.17 (pg 9.6.11) to any of 12.2.x, 12.3.x or 12.5.x (pg 10.9) it fails to start.

After countless tries and rm -R /var/opt/gitlab/postgresql/data.10 I hope anyone can help me out with this.

It won’t start due to the postgres upgrade error.

Running handlers:
Running handlers complete
Chef Client finished, 379/1345 resources updated in 01 minutes 25 seconds
gitlab Reconfigured!
Checking for an omnibus managed postgresql: OK
Checking for a newer version of PostgreSQL to install
Upgrading PostgreSQL to 10.9
Checking if we already upgraded: NOT OK
Checking if PostgreSQL bin files are symlinked to the expected location: OK
Checking if postgresql['version'] is set: OK
cp /opt/gitlab/embedded/service/gitlab-rails/public/deploy.html /opt/gitlab/embedded/service/gitlab-rails/public/index.html
Toggling deploy page: OK
Toggling services:ok: down: alertmanager: 1s, normally up
ok: down: gitaly: 0s, normally up
ok: down: gitlab-exporter: 0s, normally up
ok: down: grafana: 1s, normally up
ok: down: logrotate: 0s, normally up
ok: down: postgres-exporter: 1s, normally up
ok: down: prometheus: 0s, normally up
ok: down: redis-exporter: 0s, normally up
ok: down: sidekiq: 0s, normally up
ok: down: sshd: 1s, normally up
Toggling services: OK
Running stop on postgresql:ok: down: postgresql: 0s, normally up
Running stop on postgresql: OK
Symlink correct version of binaries: OK
Creating temporary data directory: OK
Initializing the new database: OK
Upgrading the data: NOT OK
Error upgrading the data to version 10.9
You must have read and write access in the current directory.
Failure, exiting
/opt/gitlab/embedded/service/omnibus-ctl/lib/gitlab_ctl/pg_upgrade.rb:130:in `run_pg_upgrade': undefined method `ExecutionError' for GitlabCtl::Errors:Class (NoMethodError)
	from /opt/gitlab/embedded/service/omnibus-ctl/pg-upgrade.rb:228:in `general_upgrade'
	from /opt/gitlab/embedded/service/omnibus-ctl/pg-upgrade.rb:175:in `block in load_file'
	from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/omnibus-ctl-0.6.0/lib/omnibus-ctl.rb:204:in `block in add_command_under_category'
	from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/omnibus-ctl-0.6.0/lib/omnibus-ctl.rb:746:in `run'
	from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/omnibus-ctl-0.6.0/bin/omnibus-ctl:31:in `<top (required)>'
	from /opt/gitlab/embedded/bin/omnibus-ctl:23:in `load'
	from /opt/gitlab/embedded/bin/omnibus-ctl:23:in `<main>'
Upgrading the existing database to 10 failed and was reverted.
Please check the output, and open an issue at:
If you would like to restart the instance without attempting to
upgrade, add the following to your docker command:

gitlab-omnibus config

    external_url ''
    gitlab_rails['time_zone'] = 'Europe/Berlin'
    gitlab_rails['gitlab_signup_enabled'] = false
    gitlab_rails['gitlab_signin_enabled'] = true
    gitlab_rails['gitlab_default_projects_limit'] = 100
    gitlab_rails['gravatar_enabled'] = false        
    gitlab_rails['gitlab_default_can_create_group'] = false
    gitlab_rails['gitlab_username_changing_enabled'] = false
    gitlab_rails['gitlab_default_theme'] = 2
    gitlab_rails['gitlab_default_projects_features_issues'] = false
    gitlab_rails['gitlab_default_projects_features_merge_requests'] = true
    gitlab_rails['gitlab_default_projects_features_wiki'] = false
    gitlab_rails['gitlab_default_projects_features_snippets'] = false
    gitlab_rails['gitlab_email_from'] = ''
    gitlab_rails['smtp_enable'] = true
    gitlab_rails['smtp_address'] = ''
    gitlab_rails['smtp_port'] = 465
    gitlab_rails['smtp_user_name'] = ''
    gitlab_rails['smtp_password'] = 'p@$$w0rd'
    gitlab_rails['smtp_tls'] = true
    gitlab_shell['http_settings_self_signed_cert'] = 'true'
    nginx['listen_port'] = 80
    nginx['listen_https'] = false
    nginx['proxy_set_headers'] = {
        "X-Forwarded-Proto" => "https",
        "X-Forwarded-Ssl" => "on" }

Skipping the postgres update using -e GITLAB_SKIP_PG_UPGRADE=true works but even if executed gitlab-ctl pg-upgrade afterwards, it fails to upgrade the database with the same error message.

And docker exec gitlab update-permissions did not help.

May be someone encountered something similar and can help?

Already created issue #4967 on