Gitlab is not starting up after attempting to upgrade to the latest release. It appears to be related to an issue with postgres. (Note: there are two ‘data’ directories in /var/opt/gitlab/postgresql. ‘data’ and ‘data.10’. unsure how to repair to get things running again.
Checking if PostgreSQL bin files are symlinked to the expected location: OK
Toggling deploy page:cp /opt/gitlab/embedded/service/gitlab-rails/public/deploy.html /opt/gitlab/embedded/service/gitlab-rails/public/index.html
Toggling deploy page: OK
Toggling services:ok: down: alertmanager: 17s, normally up
ok: down: gitaly: 1s, normally up
ok: down: gitlab-monitor: 17s, normally up
ok: down: grafana: 16s, normally up
ok: down: logrotate: 16s, normally up
ok: down: node-exporter: 15s, normally up
ok: down: postgres-exporter: 15s, normally up
ok: down: prometheus: 14s, normally up
ok: down: redis-exporter: 14s, normally up
ok: down: sidekiq: 14s, normally up
Toggling services: OK
Stopping the database:ok: down: postgresql: 0s, normally up
Stopping the database: OK
Symlink correct version of binaries: OK
Creating temporary data directory: OK
Initializing the new database:Error initializing database for 10.7
STDOUT: The files belonging to this database system will be owned by user “gitlab-psql”.
This user must also own the server process.
The database cluster will be initialized with locale “C.UTF-8”.
The default text search configuration will be set to “english”.
Data page checksums are disabled.
STDERR: initdb: directory “/var/opt/gitlab/postgresql/data.10” exists but is not empty
If you want to create a new database system, either remove or empty
the directory “/var/opt/gitlab/postgresql/data.10” or run initdb
with an argument other than “/var/opt/gitlab/postgresql/data.10”.
== Fatal error ==
Please check the output and try again
Traceback (most recent call last):
14: from /opt/gitlab/embedded/bin/omnibus-ctl:23:in <main>' 13: from /opt/gitlab/embedded/bin/omnibus-ctl:23:in load’
12: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/omnibus-ctl-0.6.0/bin/omnibus-ctl:31:in <top (required)>' 11: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/omnibus-ctl-0.6.0/lib/omnibus-ctl.rb:746:in run’
10: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/omnibus-ctl-0.6.0/lib/omnibus-ctl.rb:204:in block in add_command_under_category' 9: from /opt/gitlab/embedded/service/omnibus-ctl/pg-upgrade.rb:166:in block in load_file’
8: from /opt/gitlab/embedded/service/omnibus-ctl/pg-upgrade.rb:219:in general_upgrade' 7: from /opt/gitlab/embedded/service/omnibus-ctl/pg-upgrade.rb:179:in common_pre_upgrade’
6: from /opt/gitlab/embedded/service/omnibus-ctl/pg-upgrade.rb:289:in initialize_new_db' 5: from /opt/gitlab/embedded/service/omnibus-ctl/lib/gitlab_ctl/util.rb:122:in progress_message’
4: from /opt/gitlab/embedded/service/omnibus-ctl/pg-upgrade.rb:291:in block in initialize_new_db' 3: from /opt/gitlab/embedded/service/omnibus-ctl/lib/gitlab_ctl/pg_upgrade.rb:49:in run_pg_command’
2: from /opt/gitlab/embedded/service/omnibus-ctl/lib/gitlab_ctl/util.rb:14:in get_command_output' 1: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/mixlib-shellout-3.0.4/lib/mixlib/shellout.rb:284:in error!’
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/mixlib-shellout-3.0.4/lib/mixlib/shellout.rb:297:in `invalid!’: Expected process to exit with [0], but received ‘1’ (Mixlib::ShellOut::ShellCommandFailed)
---- Begin output of su - gitlab-psql -c “/opt/gitlab/embedded/bin/initdb -D /var/opt/gitlab/postgresql/data.10 --locale C.UTF-8 --encoding UTF8 --lc-collate=C.UTF-8 --lc-ctype=C.UTF-8” ----
STDOUT: The files belonging to this database system will be owned by user “gitlab-psql”.
This user must also own the server process.
The database cluster will be initialized with locale “C.UTF-8”.
The default text search configuration will be set to “english”.
Data page checksums are disabled.
STDERR: initdb: directory “/var/opt/gitlab/postgresql/data.10” exists but is not empty
If you want to create a new database system, either remove or empty
the directory “/var/opt/gitlab/postgresql/data.10” or run initdb
with an argument other than “/var/opt/gitlab/postgresql/data.10”.
---- End output of su - gitlab-psql -c “/opt/gitlab/embedded/bin/initdb -D /var/opt/gitlab/postgresql/data.10 --locale C.UTF-8 --encoding UTF8 --lc-collate=C.UTF-8 --lc-ctype=C.UTF-8” ----
Ran su - gitlab-psql -c “/opt/gitlab/embedded/bin/initdb -D /var/opt/gitlab/postgresql/data.10 --locale C.UTF-8 --encoding UTF8 --lc-collate=C.UTF-8 --lc-ctype=C.UTF-8” returned 1
13: from /opt/gitlab/embedded/bin/omnibus-ctl:23:in <main>' 12: from /opt/gitlab/embedded/bin/omnibus-ctl:23:in load’
11: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/omnibus-ctl-0.6.0/bin/omnibus-ctl:31:in <top (required)>' 10: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/omnibus-ctl-0.6.0/lib/omnibus-ctl.rb:746:in run’
9: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/omnibus-ctl-0.6.0/lib/omnibus-ctl.rb:204:in block in add_command_under_category' 8: from /opt/gitlab/embedded/service/omnibus-ctl/pg-upgrade.rb:166:in block in load_file’
7: from /opt/gitlab/embedded/service/omnibus-ctl/pg-upgrade.rb:219:in general_upgrade' 6: from /opt/gitlab/embedded/service/omnibus-ctl/pg-upgrade.rb:179:in common_pre_upgrade’
5: from /opt/gitlab/embedded/service/omnibus-ctl/pg-upgrade.rb:289:in initialize_new_db' 4: from /opt/gitlab/embedded/service/omnibus-ctl/lib/gitlab_ctl/util.rb:122:in progress_message’
3: from /opt/gitlab/embedded/service/omnibus-ctl/pg-upgrade.rb:291:in block in initialize_new_db' 2: from /opt/gitlab/embedded/service/omnibus-ctl/lib/gitlab_ctl/pg_upgrade.rb:49:in run_pg_command’
1: from /opt/gitlab/embedded/service/omnibus-ctl/lib/gitlab_ctl/util.rb:12:in get_command_output' /opt/gitlab/embedded/service/omnibus-ctl/lib/gitlab_ctl/util.rb:16:in rescue in get_command_output’: GitlabCtl::Errors::ExecutionError (GitlabCtl::Errors::ExecutionError)
12: from /opt/gitlab/embedded/bin/omnibus-ctl:23:in <main>' 11: from /opt/gitlab/embedded/bin/omnibus-ctl:23:in load’
10: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/omnibus-ctl-0.6.0/bin/omnibus-ctl:31:in <top (required)>' 9: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/omnibus-ctl-0.6.0/lib/omnibus-ctl.rb:746:in run’
8: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/omnibus-ctl-0.6.0/lib/omnibus-ctl.rb:204:in block in add_command_under_category' 7: from /opt/gitlab/embedded/service/omnibus-ctl/pg-upgrade.rb:166:in block in load_file’
6: from /opt/gitlab/embedded/service/omnibus-ctl/pg-upgrade.rb:219:in general_upgrade' 5: from /opt/gitlab/embedded/service/omnibus-ctl/pg-upgrade.rb:179:in common_pre_upgrade’
4: from /opt/gitlab/embedded/service/omnibus-ctl/pg-upgrade.rb:289:in initialize_new_db' 3: from /opt/gitlab/embedded/service/omnibus-ctl/lib/gitlab_ctl/util.rb:122:in progress_message’
2: from /opt/gitlab/embedded/service/omnibus-ctl/pg-upgrade.rb:290:in block in initialize_new_db' 1: from /opt/gitlab/embedded/service/omnibus-ctl/pg-upgrade.rb:303:in rescue in block in initialize_new_db’
/opt/gitlab/embedded/service/omnibus-ctl/lib/gitlab_ctl/pg_upgrade.rb:149:in die': undefined local variable or method revert’ for GitlabCtl::PgUpgrade:Class (NameError)
Ensuring PostgreSQL is updated: NOT OK
Error ensuring PostgreSQL is updated. Please check the logs
dpkg: error processing package gitlab-ee (–configure):
installed gitlab-ee package post-installation script subprocess returned error exit status 1
Errors were encountered while processing:
gitlab-ee
E: Sub-process /usr/bin/dpkg returned an error code (1)