Postgresql is down because of the incompatibility between data directory and postgresql database

Recently we have upgraded the gitlab 12.7.5 CE to latest version 13.6 Enterprise edition, and gitlab and postgresql upgrade automatically and data directory is still at the older version. Because of which postgresql is down and unable to start the gitlab instance.

Below is the version information.

[gitlab@host ~] **sudo /opt/gitlab/embedded/bin/psql --version** [sudo] password for gitlab: psql (PostgreSQL) 11.9 [gitlab@host ~]
[gitlab@host ~] [gitlab@host ~]
[gitlab@host ~] **sudo /opt/gitlab/embedded/bin/psql --version** psql (PostgreSQL) 11.9 [gitlab@host ~]
[gitlab@host ~] **sudo cat /var/opt/gitlab/postgresql/data/PG_VERSION** 10 [gitlab@host ~]

As you can see above, the data directory is still at 10, Want to upgrade it to the version 11.9 to start the gitlab successfully. Do you have suggestions on this?

I ran the pg-upgrade but threw below error.

[gitlab@host ~]$ sudo gitlab-ctl pg-upgrade
Traceback (most recent call last):
8: from /opt/gitlab/embedded/bin/omnibus-ctl:23:in <main>' 7: from /opt/gitlab/embedded/bin/omnibus-ctl:23:in load’
6: from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/omnibus-ctl-0.6.0/bin/omnibus-ctl:31:in <top (required)>' 5: from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/omnibus-ctl-0.6.0/lib/omnibus-ctl.rb:746:in run’
4: from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/omnibus-ctl-0.6.0/lib/omnibus-ctl.rb:204:in block in add_command_under_category' 3: from /opt/gitlab/embedded/service/omnibus-ctl/pg-upgrade.rb:142:in block in load_file’
2: from /opt/gitlab/embedded/service/omnibus-ctl/lib/gitlab_ctl/util.rb:100:in roles' 1: from /opt/gitlab/embedded/service/omnibus-ctl/lib/gitlab_ctl/util.rb:59:in get_node_attributes’
/opt/gitlab/embedded/service/omnibus-ctl/lib/gitlab_ctl/util.rb:47:in `parse_json_file’: Attributes not found in /opt/gitlab/embedded/nodes/sgpl-gitlab.sgpltech.com.json, has reconfigure been run yet? (GitlabCtl::Errors::NodeError)

I ran the gitlab reconfigure it throw the below error:

Running handlers:
There was an error running gitlab-ctl reconfigure:

bash[migrate gitlab-rails database] (gitlab::database_migrations line 55) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received ‘1’
---- Begin output of “bash” “/tmp/chef-script20201228-30383-1vop4xx” ----
STDOUT: rake aborted!
PG::ConnectionBad: could not connect to server: No such file or directory
Is the server running locally and accepting
connections on Unix domain socket “/var/opt/gitlab/postgresql/.s.PGSQL.5432”?
/opt/gitlab/embedded/service/gitlab-rails/ee/app/models/license.rb:273:in load_license' /opt/gitlab/embedded/service/gitlab-rails/ee/app/models/license.rb:258:in current’
/opt/gitlab/embedded/service/gitlab-rails/ee/app/models/license.rb:266:in feature_available?' /opt/gitlab/embedded/service/gitlab-rails/ee/lib/ee/gitlab/auth/ldap/config.rb:19:in _available_servers’
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/auth/ldap/config.rb:37:in available_servers' /opt/gitlab/embedded/service/gitlab-rails/config/initializers/omniauth.rb:3:in module:Strategies’
/opt/gitlab/embedded/service/gitlab-rails/config/initializers/omniauth.rb:2:in <top (required)>' /opt/gitlab/embedded/service/gitlab-rails/config/environment.rb:5:in <top (required)>’
/opt/gitlab/embedded/bin/bundle:23:in load' /opt/gitlab/embedded/bin/bundle:23:in ’
Tasks: TOP => gitlab:db:configure => environment
(See full trace by running task with --trace)

Anyone faced this issue before and what steps do we need to follow to make the data directory and postgresql version compatible, so that my gitlab EE will be up and running.

Thanks in advance.

I am guessing you didnt follow the proper upgrade path as per the gitlab upgrade docs? If so then restore your server and start again making sure to upgrade to all appropriate point releases before attempting to go to 13.6.

If not then please post exactly what upgrade path you chose to get from 12.7.x to 13.6.x.

But either way you will need to restore your server and start again as there are migrations that need to be run and completed from certain versions during the upgrade process and further upgrades should not be attempted until the migrations have finished. Also by jumping too far ahead with upgrades you can miss migrations and other important things.

Check the gitlab upgrade docs and the paths that it shows to upgrade to.

1 Like

How to revert back to 12.7.5 from 13.6? I tried to revert back but there is no backup available on the instance. Do you have any proper steps to revert it back to older version so that I wont lose any data or at least repos that I have?

Appreciate your help.

Before doing an upgrade you should have taken a backup. How do you think it will be possible to restore it when you don’t have a backup? Backups are the responsibility of the person who manages the server. When I do upgrades I always ensure I have backups of my servers whether it is gitlab, email, web server or whatever.

Even without upgrading you should have been making regular backups of your server in the event of server problems or data corruption.

If you have all your repositories cloned then make a new server installation, create an empty repo with the same name as your existing one and then push all your data to it from the repositories you have locally on your machine. I hope you do have them all cloned somewhere? If not then it looks like you will have lost it.

This is a test environment we recently installed and have a few repo in it. But suddenly due to yum update rpm got updated unintentionally and caused the gitlab to upgrade. Is there any way that we can revert to old 12.7.5?

Also i checked the repositories directories there are no repos are available. do you know where gitlab stores them?

[root@host git-data]# ls
repositories
[root@host git-data]# cd repositories/
[root@host repositories]# ls
+gitaly @hashed @pools
[root@host repositories]#

As I already said, without a backup, you have no chance to revert. Your repositories are now inside the @hashed directories and so it won’t be possible for you to identify them and revert to 12.7.5.

Potentially you have a backup under /var/opt/gitlab/backups of which you can restore once you are on a clean installation of 12.7.5 with the correct configuration from /etc/gitlab/gitlab.rb and providing you also have the gitlab-secrets.json file of which you can follow the gitlab docs here:

if not, then you can only do what I said before. Make a clean install of gitlab, and then with the cloned repositories locally on your laptop/desktop or wherever, and then push them to the new clean installation. As without a backup you will not be able to recover the existing server that is messed up.