PG::ConnectionBad: FATAL: the database system is shutting down

I’ve installed package “gitlab-ce” on Debian Stretch from repository “https://packages.gitlab.com/gitlab/gitlab-ce/debian/”. The installation seemed to go OK, with the Gitlab ascii art showing up, but when running gitlab-ctl reconfigure it has a fatal error form what appears to be the embedded PGSQL service.

Console output is as follows:

================================================================================
Error executing action `run` on resource 'bash[migrate gitlab-rails database]'
================================================================================

Mixlib::ShellOut::ShellCommandFailed
------------------------------------
Expected process to exit with [0], but received '1'
---- Begin output of "bash"  "/tmp/chef-script20180708-2286-qe7r6u" ----
STDOUT: rake aborted!
PG::ConnectionBad: FATAL:  the database system is shutting down
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:49:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Tasks: TOP => gitlab:db:configure
(See full trace by running task with --trace)
STDERR:
---- End output of "bash"  "/tmp/chef-script20180708-2286-qe7r6u" ----
Ran "bash"  "/tmp/chef-script20180708-2286-qe7r6u" returned 1

Resource Declaration:
---------------------
# In /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/database_migrations.rb

 49: bash "migrate gitlab-rails database" do
 50:   code <<-EOH
 51:     set -e
 52:     log_file="#{node['gitlab']['gitlab-rails']['log_directory']}/gitlab-rails-db-migrate-$(date +%Y-%m-%d-%H-%M-%S).log"
 53:     umask 077
 54:     /opt/gitlab/bin/gitlab-rake gitlab:db:configure 2>& 1 | tee ${log_file}
 55:     STATUS=${PIPESTATUS[0]}
 56:     echo $STATUS > #{db_migrate_status_file}
 57:     exit $STATUS
 58:   EOH
 59:   environment env_variables unless env_variables.empty?
 60:   notifies :run, "execute[clear the gitlab-rails cache]", :immediately
 61:   dependent_services.each do |svc|
 62:     notifies :restart, svc, :immediately
 63:   end
 64:   not_if "(test -f #{db_migrate_status_file}) && (cat #{db_migrate_status_file} | grep -Fx 0)"
 65:   only_if { node['gitlab']['gitlab-rails']['auto_migrate'] }
 66: end

If I attempt to use anything from gitlab-psql I get the same error message, psql: FATAL: the database system is shutting down.

I don’t mind deleting Gitlab CE completely as it contains no working data yet. I did try apt-get purge gitlab-ce then re-installing it, but the issue still occurs. Not sure if bad data left over somewhere or a bug in the installation.

Resolved.

After doing apt-get purge gitlab-ce you must manually remove:

  • /opt/gitlab
  • /var/opt/gitlab

I didn’t notice the 2nd one existed, only found out by chance when searching the whole file system for “pgsql”.

After removing those directories and installing gitlab-ce again the gitlab-ctl reconfigure command worked first time. I guess these directories don’t get removed with a purge by mistake.