Upgrading 11.4.14 to 11.5.11 fails

I am trying to upgrade from gitlab -ce 11.4.14 to 11.5.11, the upgrade fails because
it is unable to connect to the postgresql server (see below). Earlier today I updated from 11.0.0 ->11.1.8->11.2.8->11.3.14->11.4.14

Recipe: gitlab::database_migrations
  * bash[migrate gitlab-rails database] action run
    [execute] rake aborted!
              PG::ConnectionBad: could not connect to server: Permission denied
              	Is the server running locally and accepting
              	connections on Unix domain socket "/db/postgresql/.s.PGSQL.5432"?
              /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:51:in `block (3 levels) in <top (required)>'
              /opt/gitlab/embedded/bin/bundle:23:in `load'
              /opt/gitlab/embedded/bin/bundle:23:in `<main>'
              Tasks: TOP => gitlab:db:configure
              (See full trace by running task with --trace)
    
    ================================================================================
    Error executing action `run` on resource 'bash[migrate gitlab-rails database]'
    ================================================================================
    
    Mixlib::ShellOut::ShellCommandFailed
    ------------------------------------
    Expected process to exit with [0], but received '1'
    ---- Begin output of "bash"  "/tmp/chef-script20200625-40603-10sld8g" ----
    STDOUT: rake aborted!
    PG::ConnectionBad: could not connect to server: Permission denied
    	Is the server running locally and accepting
    	connections on Unix domain socket "/db/postgresql/.s.PGSQL.5432"?
    /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:51:in `block (3 levels) in <top (required)>'
    /opt/gitlab/embedded/bin/bundle:23:in `load'
    /opt/gitlab/embedded/bin/bundle:23:in `<main>'
    Tasks: TOP => gitlab:db:configure
    (See full trace by running task with --trace)
    STDERR: 
    ---- End output of "bash"  "/tmp/chef-script20200625-40603-10sld8g" ----
    Ran "bash"  "/tmp/chef-script20200625-40603-10sld8g" returned 1
    
    Resource Declaration:
    ---------------------
    # In /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/database_migrations.rb
    
     50: bash "migrate gitlab-rails database" do
     51:   code <<-EOH
     52:     set -e
     53:     log_file="#{node['gitlab']['gitlab-rails']['log_directory']}/gitlab-rails-db-migrate-$(date +%Y-%m-%d-%H-%M-%S).log"
     54:     umask 077
     55:     /opt/gitlab/bin/gitlab-rake gitlab:db:configure 2>& 1 | tee ${log_file}
     56:     STATUS=${PIPESTATUS[0]}
     57:     echo $STATUS > #{db_migrate_status_file}
     58:     exit $STATUS
     59:   EOH
     60:   environment env_variables unless env_variables.empty?
     61:   notifies :run, "execute[clear the gitlab-rails cache]", :immediately
     62:   dependent_services.each do |svc|
     63:     notifies :restart, svc, :immediately
     64:   end
     65:   not_if "(test -f #{db_migrate_status_file}) && (cat #{db_migrate_status_file} | grep -Fx 0)"
     66:   only_if { node['gitlab']['gitlab-rails']['auto_migrate'] }
     67: end
    
    Compiled Resource:
    ------------------
    # Declared in /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/database_migrations.rb:50:in `from_file'
    
    bash("migrate gitlab-rails database") do
      action [:run]
      default_guard_interpreter :default
      command nil
      backup 5
      returns 0
      user nil
      interpreter "bash"
      declared_type :bash
      cookbook_name "gitlab"
      recipe_name "database_migrations"
      code "    set -e\n    log_file=\"/var/log/gitlab/gitlab-rails/gitlab-rails-db-migrate-$(date +%Y-%m-%d-%H-%M-%S).log\"\n    umask 077\n    /opt/gitlab/bin/gitlab-rake gitlab:db:configure 2>& 1 | tee ${log_file}\n    STATUS=${PIPESTATUS[0]}\n    echo $STATUS > /var/opt/gitlab/gitlab-rails/upgrade-status/db-migrate-77e55c568c4b132cc84972b2648b48c3-e9a219a\n    exit $STATUS\n"
      domain nil
      not_if "(test -f /var/opt/gitlab/gitlab-rails/upgrade-status/db-migrate-77e55c568c4b132cc84972b2648b48c3-e9a219a) && (cat /var/opt/gitlab/gitlab-rails/upgrade-status/db-migrate-77e55c568c4b132cc84972b2648b48c3-e9a219a | grep -Fx 0)"
      only_if { #code block }
    end
    
    System Info:
    ------------
    chef_version=13.6.4
    platform=debian
    platform_version=8.11
    ruby=ruby 2.4.5p335 (2018-10-18 revision 65137) [x86_64-linux]
    program_name=/opt/gitlab/embedded/bin/chef-client
    executable=/opt/gitlab/embedded/bin/chef-client
    

Running handlers:
There was an error running gitlab-ctl reconfigure:

bash[migrate gitlab-rails database] (gitlab::database_migrations line 50) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
---- Begin output of "bash"  "/tmp/chef-script20200625-40603-10sld8g" ----
STDOUT: rake aborted!
PG::ConnectionBad: could not connect to server: Permission denied
	Is the server running locally and accepting
	connections on Unix domain socket "/db/postgresql/.s.PGSQL.5432"?
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:51:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Tasks: TOP => gitlab:db:configure
(See full trace by running task with --trace)
STDERR: 
---- End output of "bash"  "/tmp/chef-script20200625-40603-10sld8g" ----
Ran "bash"  "/tmp/chef-script20200625-40603-10sld8g" returned 1

Hi and welcome to our community :slight_smile:

Are you running GitLab in a Docker container? The error sounds a bit like that. If that is the case, you might want to try to fix the permissions first and then try to restart, following the upgrade.

Cheers,
Michael

Iā€™m not running gitlab in a container, gitlab-ctl status reports

run: alertmanager: (pid 11518) 9749s; run: log: (pid 7473) 19750s
run: gitaly: (pid 45916) 11113s; run: log: (pid 7479) 19750s
run: gitlab-monitor: (pid 11546) 9748s; run: log: (pid 7482) 19750s
run: gitlab-workhorse: (pid 11551) 9748s; run: log: (pid 7488) 19750s
run: logrotate: (pid 930) 2547s; run: log: (pid 7484) 19750s
run: mattermost: (pid 11574) 9747s; run: log: (pid 7489) 19750s
run: node-exporter: (pid 11607) 9747s; run: log: (pid 7477) 19750s
run: postgres-exporter: (pid 11613) 9746s; run: log: (pid 7486) 19750s
run: postgresql: (pid 26795) 9005s; run: log: (pid 7475) 19750s
run: prometheus: (pid 11624) 9746s; run: log: (pid 7478) 19750s
run: redis: (pid 11505) 9749s; run: log: (pid 7476) 19750s
run: redis-exporter: (pid 11658) 9746s; run: log: (pid 7472) 19750s
run: sidekiq: (pid 48156) 5s; run: log: (pid 7480) 19750s
run: unicorn: (pid 48216) 1s; run: log: (pid 7487) 19750s

and the socket exists

ls -l ā€œ/db/postgresql/.s.PGSQL.5432ā€
srwxrwxrwx 1 gitlab-psql gitlab-psql 0 Jun 25 14:29 /db/postgresql/.s.PGSQL.5432

Thanks

David