I have a gitlab instance installed on a centos 7.8 VM. I was having some issues renewing the LE certs via gitlab-ctl reconfigure and noticed a forum suggested upgrading may help - so I ran yum -y update and rebooted - gitlab appeared to come back online fine initially, however it looks like postgresql starts fine, but dies immediately:
[root@GITLAB ~]# gitlab-ctl status
run: alertmanager: (pid 21393) 936s; run: log: (pid 9834) 3492s
run: crond: (pid 21407) 936s; run: log: (pid 9843) 3492s
run: gitaly: (pid 21420) 935s; run: log: (pid 9823) 3492s
run: gitlab-exporter: (pid 21430) 935s; run: log: (pid 9835) 3492s
run: gitlab-workhorse: (pid 21441) 935s; run: log: (pid 9809) 3492s
run: grafana: (pid 21454) 934s; run: log: (pid 9846) 3492s
run: logrotate: (pid 21470) 934s; run: log: (pid 9822) 3492s
run: nginx: (pid 21477) 934s; run: log: (pid 9818) 3493s
run: node-exporter: (pid 21493) 934s; run: log: (pid 9813) 3493s
run: postgres-exporter: (pid 21498) 933s; run: log: (pid 9837) 3493s
**down: postgresql: 1s, normally up, want up; run: log: (pid 9833) 3493s**
run: prometheus: (pid 21614) 927s; run: log: (pid 9825) 3493s
run: redis: (pid 21624) 926s; run: log: (pid 9831) 3493s
run: redis-exporter: (pid 21630) 926s; run: log: (pid 9824) 3493s
run: sidekiq: (pid 25246) 3s; run: log: (pid 9832) 3493s
run: unicorn: (pid 25221) 7s; run: log: (pid 9806) 3493s
The logs suggest there is a version incompatibility:
[root@GITLAB ~]# gitlab-ctl tail postgresql
==> /var/log/gitlab/postgresql/state <==
==> /var/log/gitlab/postgresql/current <==
2020-05-29_11:59:52.94849 FATAL: database files are incompatible with server
2020-05-29_11:59:52.94850 DETAIL: The data directory was initialized by PostgreSQL version 10, which is not compatible with this version 11.7.
2020-05-29_11:59:53.97221 FATAL: database files are incompatible with server
2020-05-29_11:59:53.97222 DETAIL: The data directory was initialized by PostgreSQL version 10, which is not compatible with this version 11.7.
2020-05-29_11:59:55.00948 FATAL: database files are incompatible with server
[root@GITLAB ~]# cat /var/opt/gitlab/postgresql/data/PG_VERSION
10
[root@GITLAB ~]# /opt/gitlab/embedded/bin/pg_ctl --version
pg_ctl (PostgreSQL) 11.7
[root@GITLAB ~]# /opt/gitlab/embedded/bin/psql --version
psql (PostgreSQL) 11.7
I’ve read some articles suggesting that running gitlab-ctl pg-upgrade could fix this however this prompts for gitlab-ctl reconfigure to be run:
gitlab-ctl pg-upgrade
[root@GITLAB ~]# gitlab-ctl pg-upgrade
Traceback (most recent call last):
8: from /opt/gitlab/embedded/bin/omnibus-ctl:23:in `<main>'
7: from /opt/gitlab/embedded/bin/omnibus-ctl:23:in `load'
6: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/omnibus-ctl-0.6.0/bin/omnibus-ctl:31:in `<top (required)>'
5: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/omnibus-ctl-0.6.0/lib/omnibus-ctl.rb:746:in `run'
4: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/omnibus-ctl-0.6.0/lib/omnibus-ctl.rb:204:in `block in add_command_under_category'
3: from /opt/gitlab/embedded/service/omnibus-ctl/pg-upgrade.rb:115:in `block in load_file'
2: from /opt/gitlab/embedded/service/omnibus-ctl/lib/gitlab_ctl/util.rb:105:in `roles'
1: from /opt/gitlab/embedded/service/omnibus-ctl/lib/gitlab_ctl/util.rb:64:in `get_node_attributes'
/opt/gitlab/embedded/service/omnibus-ctl/lib/gitlab_ctl/util.rb:52:in `parse_json_file': Attributes not found in /opt/gitlab/embedded/nodes/gitlab.mydomain.com.json, has reconfigure been run yet? (GitlabCtl::Errors::NodeError)
when I run gitlab-ctl reconfigure it errors due to postgresql being unable to connect:
gitlab-ctl reconfigure
…
Compiled Resource:
------------------
# Declared in /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/database_migrations.rb:55:in `from_file'
bash("migrate gitlab-rails database") do
action [:run]
default_guard_interpreter :default
command nil
backup 5
interpreter "bash"
declared_type :bash
cookbook_name "gitlab"
recipe_name "database_migrations"
code " set -e\n log_file=\"/var/log/gitlab/gitlab-rails/gitlab-rails-db-migrate-$(date +%Y-%m-%d-%H-%M-%S).log\"\n umask 077\n /opt/gitlab/bin/gitlab-rake gitlab:db:configure 2>& 1 | tee ${log_file}\n STATUS=${PIPESTATUS[0]}\n chown git:git ${log_file}\n echo $STATUS > /var/opt/gitlab/gitlab-rails/upgrade-status/db-migrate-873248b1f0d3a7a5535771a3a1635803-74623c80da9\n exit $STATUS\n"
domain nil
user nil
not_if "(test -f /var/opt/gitlab/gitlab-rails/upgrade-status/db-migrate-873248b1f0d3a7a5535771a3a1635803-74623c80da9) && (cat /var/opt/gitlab/gitlab-rails/upgrade-status/db-migrate-873248b1f0d3a7a5535771a3a1635803-74623c80da9 | grep -Fx 0)"
only_if { #code block }
end
System Info:
------------
chef_version=14.14.29
platform=centos
platform_version=7.8.2003
ruby=ruby 2.6.6p146 (2020-03-31 revision 67876) [x86_64-linux]
program_name=/opt/gitlab/embedded/bin/chef-client
executable=/opt/gitlab/embedded/bin/chef-client
Running handlers:
There was an error running gitlab-ctl reconfigure:
bash[migrate gitlab-rails database] (gitlab::database_migrations line 55) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
---- Begin output of "bash" "/tmp/chef-script20200529-31793-1fru7vk" ----
STDOUT: rake aborted!
PG::ConnectionBad: could not connect to server: No such file or directory
Is the server running locally and accepting
connections on Unix domain socket "/var/opt/gitlab/postgresql/.s.PGSQL.5432"?
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:48:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Tasks: TOP => gitlab:db:configure
(See full trace by running task with --trace)
STDERR:
---- End output of "bash" "/tmp/chef-script20200529-31793-1fru7vk" ----
Ran "bash" "/tmp/chef-script20200529-31793-1fru7vk" returned 1
Running handlers complete
Chef Client failed. 4 resources updated in 01 minutes 04 seconds
I seem to be going in a loop trying to figure this out. I guess the best way forward is to manually downgrade postgresql? In which case is there any documentation available for doing so?
NB: gitlab-ctl revert-pg-upgrade doesn’t work since 13.01 is shipped with 11.7:
[root@PRODC-GITLAB-SLO1 ~]# gitlab-ctl revert-pg-upgrade -V 10
The specified major version 10 is not supported. Choose from one of 11, 11, 11.
[root@PRODC-GITLAB-SLO1 ~]# gitlab-ctl revert-pg-upgrade
Checking if we need to downgrade: OK
Already running 11.7
Any help is greatly appreciated!
I’ve ommited some logs and config files due to a character limit - if there’s anything specific that would help investigate this please let me know
Wish I stumbled upon this earlier!