Upgrading 11.4.14 to 11.5.11 fails

I am trying to upgrade from gitlab -ce 11.4.14 to 11.5.11, the upgrade fails because
it is unable to connect to the postgresql server (see below). Earlier today I updated from 11.0.0 ->11.1.8->11.2.8->11.3.14->11.4.14

Recipe: gitlab::database_migrations
  * bash[migrate gitlab-rails database] action run
    [execute] rake aborted!
              PG::ConnectionBad: could not connect to server: Permission denied
              	Is the server running locally and accepting
              	connections on Unix domain socket "/db/postgresql/.s.PGSQL.5432"?
              /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:51:in `block (3 levels) in <top (required)>'
              /opt/gitlab/embedded/bin/bundle:23:in `load'
              /opt/gitlab/embedded/bin/bundle:23:in `<main>'
              Tasks: TOP => gitlab:db:configure
              (See full trace by running task with --trace)
    
    ================================================================================
    Error executing action `run` on resource 'bash[migrate gitlab-rails database]'
    ================================================================================
    
    Mixlib::ShellOut::ShellCommandFailed
    ------------------------------------
    Expected process to exit with [0], but received '1'
    ---- Begin output of "bash"  "/tmp/chef-script20200625-40603-10sld8g" ----
    STDOUT: rake aborted!
    PG::ConnectionBad: could not connect to server: Permission denied
    	Is the server running locally and accepting
    	connections on Unix domain socket "/db/postgresql/.s.PGSQL.5432"?
    /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:51:in `block (3 levels) in <top (required)>'
    /opt/gitlab/embedded/bin/bundle:23:in `load'
    /opt/gitlab/embedded/bin/bundle:23:in `<main>'
    Tasks: TOP => gitlab:db:configure
    (See full trace by running task with --trace)
    STDERR: 
    ---- End output of "bash"  "/tmp/chef-script20200625-40603-10sld8g" ----
    Ran "bash"  "/tmp/chef-script20200625-40603-10sld8g" returned 1
    
    Resource Declaration:
    ---------------------
    # In /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/database_migrations.rb
    
     50: bash "migrate gitlab-rails database" do
     51:   code <<-EOH
     52:     set -e
     53:     log_file="#{node['gitlab']['gitlab-rails']['log_directory']}/gitlab-rails-db-migrate-$(date +%Y-%m-%d-%H-%M-%S).log"
     54:     umask 077
     55:     /opt/gitlab/bin/gitlab-rake gitlab:db:configure 2>& 1 | tee ${log_file}
     56:     STATUS=${PIPESTATUS[0]}
     57:     echo $STATUS > #{db_migrate_status_file}
     58:     exit $STATUS
     59:   EOH
     60:   environment env_variables unless env_variables.empty?
     61:   notifies :run, "execute[clear the gitlab-rails cache]", :immediately
     62:   dependent_services.each do |svc|
     63:     notifies :restart, svc, :immediately
     64:   end
     65:   not_if "(test -f #{db_migrate_status_file}) && (cat #{db_migrate_status_file} | grep -Fx 0)"
     66:   only_if { node['gitlab']['gitlab-rails']['auto_migrate'] }
     67: end
    
    Compiled Resource:
    ------------------
    # Declared in /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/database_migrations.rb:50:in `from_file'
    
    bash("migrate gitlab-rails database") do
      action [:run]
      default_guard_interpreter :default
      command nil
      backup 5
      returns 0
      user nil
      interpreter "bash"
      declared_type :bash
      cookbook_name "gitlab"
      recipe_name "database_migrations"
      code "    set -e\n    log_file=\"/var/log/gitlab/gitlab-rails/gitlab-rails-db-migrate-$(date +%Y-%m-%d-%H-%M-%S).log\"\n    umask 077\n    /opt/gitlab/bin/gitlab-rake gitlab:db:configure 2>& 1 | tee ${log_file}\n    STATUS=${PIPESTATUS[0]}\n    echo $STATUS > /var/opt/gitlab/gitlab-rails/upgrade-status/db-migrate-77e55c568c4b132cc84972b2648b48c3-e9a219a\n    exit $STATUS\n"
      domain nil
      not_if "(test -f /var/opt/gitlab/gitlab-rails/upgrade-status/db-migrate-77e55c568c4b132cc84972b2648b48c3-e9a219a) && (cat /var/opt/gitlab/gitlab-rails/upgrade-status/db-migrate-77e55c568c4b132cc84972b2648b48c3-e9a219a | grep -Fx 0)"
      only_if { #code block }
    end
    
    System Info:
    ------------
    chef_version=13.6.4
    platform=debian
    platform_version=8.11
    ruby=ruby 2.4.5p335 (2018-10-18 revision 65137) [x86_64-linux]
    program_name=/opt/gitlab/embedded/bin/chef-client
    executable=/opt/gitlab/embedded/bin/chef-client
    

Running handlers:
There was an error running gitlab-ctl reconfigure:

bash[migrate gitlab-rails database] (gitlab::database_migrations line 50) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
---- Begin output of "bash"  "/tmp/chef-script20200625-40603-10sld8g" ----
STDOUT: rake aborted!
PG::ConnectionBad: could not connect to server: Permission denied
	Is the server running locally and accepting
	connections on Unix domain socket "/db/postgresql/.s.PGSQL.5432"?
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:51:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Tasks: TOP => gitlab:db:configure
(See full trace by running task with --trace)
STDERR: 
---- End output of "bash"  "/tmp/chef-script20200625-40603-10sld8g" ----
Ran "bash"  "/tmp/chef-script20200625-40603-10sld8g" returned 1

Hi and welcome to our community :slight_smile:

Are you running GitLab in a Docker container? The error sounds a bit like that. If that is the case, you might want to try to fix the permissions first and then try to restart, following the upgrade.

Cheers,
Michael

I’m not running gitlab in a container, gitlab-ctl status reports

run: alertmanager: (pid 11518) 9749s; run: log: (pid 7473) 19750s
run: gitaly: (pid 45916) 11113s; run: log: (pid 7479) 19750s
run: gitlab-monitor: (pid 11546) 9748s; run: log: (pid 7482) 19750s
run: gitlab-workhorse: (pid 11551) 9748s; run: log: (pid 7488) 19750s
run: logrotate: (pid 930) 2547s; run: log: (pid 7484) 19750s
run: mattermost: (pid 11574) 9747s; run: log: (pid 7489) 19750s
run: node-exporter: (pid 11607) 9747s; run: log: (pid 7477) 19750s
run: postgres-exporter: (pid 11613) 9746s; run: log: (pid 7486) 19750s
run: postgresql: (pid 26795) 9005s; run: log: (pid 7475) 19750s
run: prometheus: (pid 11624) 9746s; run: log: (pid 7478) 19750s
run: redis: (pid 11505) 9749s; run: log: (pid 7476) 19750s
run: redis-exporter: (pid 11658) 9746s; run: log: (pid 7472) 19750s
run: sidekiq: (pid 48156) 5s; run: log: (pid 7480) 19750s
run: unicorn: (pid 48216) 1s; run: log: (pid 7487) 19750s

and the socket exists

ls -l ā€œ/db/postgresql/.s.PGSQL.5432ā€
srwxrwxrwx 1 gitlab-psql gitlab-psql 0 Jun 25 14:29 /db/postgresql/.s.PGSQL.5432

Thanks

David