Upgrade failed - Postgresql down


#1

I was updating my gitlab installation from 10.8.7 to 11.4.5 but the process was killed due to timeout.
Now I cannot star anymore gitlab.
It looks that reconfigure was not completed because teh postgresql was down:

Recipe: postgresql::enable
  * template[/opt/gitlab/etc/gitlab-psql-rc] action create (up to date)
  * postgresql_user[gitlab] action create
    * execute[create gitlab postgresql user] action run (skipped due to not_if)
     (up to date)
  * execute[create gitlabhq_production database] action run (skipped due to not_if)
  * postgresql_user[gitlab_replicator] action create
    * execute[create gitlab_replicator postgresql user] action run (skipped due to not_if)
    * execute[set options for gitlab_replicator postgresql user] action run (skipped due to not_if)
     (up to date)
  * postgresql_extension[pg_trgm] action enable
    * postgresql_query[enable pg_trgm extension] action run (skipped due to only_if)
     (up to date)
  * execute[reload postgresql] action nothing (skipped due to action :nothing)
  * execute[start postgresql] action nothing (skipped due to action :nothing)
Recipe: gitlab::database_migrations
  * bash[migrate gitlab-rails database] action run
    [execute] rake aborted!
              PG::ConnectionBad: could not connect to server: No such file or directory
                Is the server running locally and accepting
                connections on Unix domain socket "/var/opt/gitlab/postgresql/.s.PGSQL.5432"?
              /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:51:in `block (3 levels) in <top (required)>'
              /opt/gitlab/embedded/bin/bundle:23:in `load'
              /opt/gitlab/embedded/bin/bundle:23:in `<main>'
              Tasks: TOP => gitlab:db:configure
              (See full trace by running task with --trace)

    ================================================================================
    Error executing action `run` on resource 'bash[migrate gitlab-rails database]'
    ================================================================================

    Mixlib::ShellOut::ShellCommandFailed
    ------------------------------------
    Expected process to exit with [0], but received '1'
    ---- Begin output of "bash"  "/tmp/chef-script20181119-2488-7xexy2" ----
    STDOUT: rake aborted!
    PG::ConnectionBad: could not connect to server: No such file or directory
        Is the server running locally and accepting
        connections on Unix domain socket "/var/opt/gitlab/postgresql/.s.PGSQL.5432"?
    /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:51:in `block (3 levels) in <top (required)>'
    /opt/gitlab/embedded/bin/bundle:23:in `load'
    /opt/gitlab/embedded/bin/bundle:23:in `<main>'
    Tasks: TOP => gitlab:db:configure
    (See full trace by running task with --trace)
    STDERR:
    ---- End output of "bash"  "/tmp/chef-script20181119-2488-7xexy2" ----
    Ran "bash"  "/tmp/chef-script20181119-2488-7xexy2" returned 1

    Resource Declaration:
    ---------------------
    # In /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/database_migrations.rb

     49: bash "migrate gitlab-rails database" do
     50:   code <<-EOH
     51:     set -e
     52:     log_file="#{node['gitlab']['gitlab-rails']['log_directory']}/gitlab-rails-db-migrate-$(date +%Y-%m-%d-%H-%M-%S).log"
     53:     umask 077
     54:     /opt/gitlab/bin/gitlab-rake gitlab:db:configure 2>& 1 | tee ${log_file}
     55:     STATUS=${PIPESTATUS[0]}
     56:     echo $STATUS > #{db_migrate_status_file}
     57:     exit $STATUS
     58:   EOH
     59:   environment env_variables unless env_variables.empty?
     60:   notifies :run, "execute[clear the gitlab-rails cache]", :immediately
     61:   dependent_services.each do |svc|
     62:     notifies :restart, svc, :immediately
     63:   end
     64:   not_if "(test -f #{db_migrate_status_file}) && (cat #{db_migrate_status_file} | grep -Fx 0)"
     65:   only_if { node['gitlab']['gitlab-rails']['auto_migrate'] }
     66: end

    Compiled Resource:
    ------------------
    # Declared in /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/database_migrations.rb:49:in `from_file'

    bash("migrate gitlab-rails database") do
      action [:run]
      default_guard_interpreter :default
      command nil
      backup 5
      returns 0
      user nil
      interpreter "bash"
      declared_type :bash
      cookbook_name "gitlab"
      recipe_name "database_migrations"
      code "    set -e\n    log_file=\"/var/log/gitlab/gitlab-rails/gitlab-rails-db-migrate-$(date +%Y-%m-%d-%H-%M-%S).log\"\n    umask 077\n    /opt/gitlab/bin/gitlab-rake gitlab:db:configure 2>& 1 | tee ${log_file}\n    STATUS=${PIPESTATUS[0]}\n    echo $STATUS > /var/opt/gitlab/gitlab-rails/upgrade-status/db-migrate-873248b1f0d3a7a5535771a3a1635803-f5536c6\n    exit $STATUS\n"
      domain nil
      not_if "(test -f /var/opt/gitlab/gitlab-rails/upgrade-status/db-migrate-873248b1f0d3a7a5535771a3a1635803-f5536c6) && (cat /var/opt/gitlab/gitlab-rails/upgrade-status/db-migrate-873248b1f0d3a7a5535771a3a1635803-f5536c6 | grep -Fx 0)"
      only_if { #code block }
    end

    System Info:
    ------------
    chef_version=13.6.4
    platform=redhat
    platform_version=7.4
    ruby=ruby 2.4.5p335 (2018-10-18 revision 65137) [x86_64-linux]
    program_name=/opt/gitlab/embedded/bin/chef-client
    executable=/opt/gitlab/embedded/bin/chef-client


Running handlers:
There was an error running gitlab-ctl reconfigure:

bash[migrate gitlab-rails database] (gitlab::database_migrations line 49) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
---- Begin output of "bash"  "/tmp/chef-script20181119-2488-7xexy2" ----
STDOUT: rake aborted!
PG::ConnectionBad: could not connect to server: No such file or directory
        Is the server running locally and accepting
        connections on Unix domain socket "/var/opt/gitlab/postgresql/.s.PGSQL.5432"?
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:51:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Tasks: TOP => gitlab:db:configure
(See full trace by running task with --trace)
STDERR:
---- End output of "bash"  "/tmp/chef-script20181119-2488-7xexy2" ----
Ran "bash"  "/tmp/chef-script20181119-2488-7xexy2" returned 1


Deprecations:
== Prometheus ==
Detected Prometheus version 1.x. Version 1.x has been deprecated and support will be removed in GitLab version 12.0.
To upgrade to Promtheus 2.x, use `gitlab-ctl prometheus-upgrade` command.
Running this command will migrate all your existing data to format supported by Prometheus 2.x.
This can be a time consuming operation. To skip migrating the data, and instead remove and start fresh, run `gitlab-ctl prometheus-upgrade --skip-data-migration`.
Check https://docs.gitlab.com/omnibus/update/gitlab_11_changes.html#11-4 for details.
---

Running handlers complete
Chef Client failed. 5 resources updated in 01 minutes 11 seconds

Deprecations:
== Prometheus ==
Detected Prometheus version 1.x. Version 1.x has been deprecated and support will be removed in GitLab version 12.0.
To upgrade to Promtheus 2.x, use `gitlab-ctl prometheus-upgrade` command.
Running this command will migrate all your existing data to format supported by Prometheus 2.x.
This can be a time consuming operation. To skip migrating the data, and instead remove and start fresh, run `gitlab-ctl prometheus-upgrade --skip-data-migration`.
Check https://docs.gitlab.com/omnibus/update/gitlab_11_changes.html#11-4 for details.
---

I need some help to fix this problem because I ma quite new in this.

Thank you for your support