Failed to upgrade GitLab EE from 12.10.14 to 13.0.14:
apt install gitlab-ee=13.0.14*
gitlab-ctl reconfigure failed with:
There was an error running gitlab-ctl reconfigure:
bash[migrate gitlab-rails database] (gitlab::database_migrations line 55) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
---- Begin output of "bash" "/tmp/chef-script20210331-5096-c7r1k1" ----
STDOUT: rake aborted!
PG::ConnectionBad: could not connect to server: No such file or directory
Is the server running locally and accepting
connections on Unix domain socket "/var/opt/gitlab/postgresql/.s.PGSQL.5432"?
/opt/gitlab/embedded/service/gitlab-rails/ee/app/models/license.rb:263:in `load_license'
/opt/gitlab/embedded/service/gitlab-rails/ee/app/models/license.rb:252:in `current'
/opt/gitlab/embedded/service/gitlab-rails/ee/app/models/license.rb:256:in `feature_available?'
/opt/gitlab/embedded/service/gitlab-rails/ee/lib/ee/gitlab/auth/ldap/config.rb:19:in `_available_servers'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/auth/ldap/config.rb:37:in `available_servers'
/opt/gitlab/embedded/service/gitlab-rails/config/initializers/omniauth.rb:3:in `<module:Strategies>'
/opt/gitlab/embedded/service/gitlab-rails/config/initializers/omniauth.rb:2:in `<top (required)>'
/opt/gitlab/embedded/service/gitlab-rails/config/environment.rb:5:in `<top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Tasks: TOP => gitlab:db:configure => environment
(See full trace by running task with --trace)
STDERR:
---- End output of "bash" "/tmp/chef-script20210331-5096-c7r1k1" ----
Ran "bash" "/tmp/chef-script20210331-5096-c7r1k1" returned 1
I tried downgrading back to 12.10.14 but it doesn’t work now:
apt install gitlab-ee=12.10.14*
Reading package lists... Done
Building dependency tree
Reading state information... Done
Selected version '12.10.14-ee.0' (gitlab-ee:1/bionic [amd64]) for 'gitlab-ee'
The following package was automatically installed and is no longer required:
daemon
Use 'sudo apt autoremove' to remove it.
The following packages will be DOWNGRADED:
gitlab-ee
0 upgraded, 0 newly installed, 1 downgraded, 0 to remove and 5 not upgraded.
Need to get 0 B/878 MB of archives.
After this operation, 216 MB of additional disk space will be used.
Do you want to continue? [Y/n] y
dpkg: warning: downgrading gitlab-ee from 13.0.14-ee.0 to 12.10.14-ee.0
(Reading database ... 246539 files and directories currently installed.)
Preparing to unpack .../gitlab-ee_12.10.14-ee.0_amd64.deb ...
Malformed configuration JSON file found at /opt/gitlab/embedded/nodes/gitlab.hostname.json.
This usually happens when your last run of `gitlab-ctl reconfigure` didn't complete successfully.
This file is used to check if any of the unsupported configurations are enabled,
and hence require a working reconfigure before upgrading.
Please run `sudo gitlab-ctl reconfigure` to fix it and try again.
dpkg: error processing archive /var/cache/apt/archives/gitlab-ee_12.10.14-ee.0_amd64.deb (--unpack):
new gitlab-ee package pre-installation script subprocess returned error exit status 1
Errors were encountered while processing:
/var/cache/apt/archives/gitlab-ee_12.10.14-ee.0_amd64.deb
E: Sub-process /usr/bin/dpkg returned an error code (1)
Hi
the first error complains that it cannot connect to postresql database. You can check if it’s running and listening and it is not blocked by firewall.
Looks like I should have performed SQL DB version migration before GitLab upgrade:
tail /var/log/gitlab/postgresql/current
2021-03-31_12:20:48.53061 FATAL: database files are incompatible with server
2021-03-31_12:20:48.53063 DETAIL: The data directory was initialized by PostgreSQL version 9.6, which is not compatible with this version 11.7.
2021-03-31_12:20:49.53581 FATAL: database files are incompatible with server
12.10 should have already upgrade your postgresql to 11.7 unless you have disabled it by creating /etc/gitlab/disable-postgresql-upgrade. I suggest to remove the file if it’s there.
Now you have to upgrade Postgresql from 9.6 to 11.7 manually.
gitlab-ctl pg-upgrade
Traceback (most recent call last):
8: from /opt/gitlab/embedded/bin/omnibus-ctl:23:in `<main>'
7: from /opt/gitlab/embedded/bin/omnibus-ctl:23:in `load'
6: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/omnibus-ctl-0.6.0/bin/omnibus-ctl:31:in `<top (required)>'
5: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/omnibus-ctl-0.6.0/lib/omnibus-ctl.rb:746:in `run'
4: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/omnibus-ctl-0.6.0/lib/omnibus-ctl.rb:204:in `block in add_command_under_category'
3: from /opt/gitlab/embedded/service/omnibus-ctl/pg-upgrade.rb:115:in `block in load_file'
2: from /opt/gitlab/embedded/service/omnibus-ctl/lib/gitlab_ctl/util.rb:105:in `roles'
1: from /opt/gitlab/embedded/service/omnibus-ctl/lib/gitlab_ctl/util.rb:64:in `get_node_attributes'
/opt/gitlab/embedded/service/omnibus-ctl/lib/gitlab_ctl/util.rb:52:in `parse_json_file': Attributes not found in /opt/gitlab/embedded/nodes/gitlab.trei.gmbh.json, has reconfigure been run yet? (GitlabCtl::Errors::NodeError)
To upgrade the PostgreSQL version, be sure that:
You’re running the latest version of GitLab, and it’s working.
If you recently upgraded, you ran sudo gitlab-ctl reconfigure successfully before you proceed.
I don’t think the tools coming with GitLab will work as they expect a working instance. I guess your only chance is to do the PG data upgrade outside of GitLab using other running Postgresql instances.
Based on your back/recovery scenarios it might be faster just to restore.
If you have an option to run a Docker image on the server (or somewhere else) I would use this image Docker Hub to upgrade the PGSQL data to 11.7 which should make GitLab happy.
Just reinstalled GitLab to 12.10.14 and didn’t even have to perform restore. GitLab just picked up the existing (old) database – I guess, it was not broken.
The database wasn’t broken, but Postgresql datafiles are incompatible between major releases. That’s why it was complaining that the database is from version 9.6 and you are trying to use it with 11. In such case Postresql won’t start and won’t modify any existing files.
And because Postgresql didn’t start, GitLab was complaining it cannot connect to it.
I suggest to enable automatic Postgresql update in GitLab and let it upgrade to 11.7. Then you can upgrade GitLab to 13.x
Thanks, @balonik , I get it now. The thing is, I didn’t disable the automatic update of PSQL: file /etc/gitlab/disable-postgresql-upgrade doesn’t exist. However, GitLab update at some previous release update could have asked if I want to upgrade & migrate DB, and I probably said ‘No’, to be on a safe side.