GitLab CE Docker fails upgrade from 15.11.13 -> 16 (after workaround)

How to Use GitLab Upgrade
1

Hello,
I’m trying to update my long-running GitLab CE instance (on Docker) from 15.11.13 to 16.0.0.
Despite following the upgrade instructions, I’m getting the following errors:

Recipe: gitlab::database_migrations
  * ruby_block[check remote PG version] action nothing (skipped due to action :nothing)
  * rails_migration[gitlab-rails] action run[2023-10-29T15:43:30+02:00] WARN: gitlab-rails does not have a log_group or default logdir mode defined. Setting to 0700.

    * bash_hide_env[migrate gitlab-rails database] action run
      [execute] rake aborted!
                ActiveRecord::ConnectionNotEstablished: could not connect to server: Connection refused
                        Is the server running locally and accepting
                        connections on Unix domain socket "/var/opt/gitlab/postgresql/.s.PGSQL.5432"?
                /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:110:in `read_write'
                /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:126:in `write_using_load_balancer'
                /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:96:in `method_missing'
                /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:115:in `configure_database'
                /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:101:in `block (3 levels) in <top (required)>'
                /opt/gitlab/embedded/bin/bundle:25:in `load'
                /opt/gitlab/embedded/bin/bundle:25:in `<main>'

                Caused by:
                PG::ConnectionBad: could not connect to server: Connection refused
                        Is the server running locally and accepting
                        connections on Unix domain socket "/var/opt/gitlab/postgresql/.s.PGSQL.5432"?
                /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:110:in `read_write'
                /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:126:in `write_using_load_balancer'
                /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:96:in `method_missing'
                /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:115:in `configure_database'
                /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:101:in `block (3 levels) in <top (required)>'
                /opt/gitlab/embedded/bin/bundle:25:in `load'
                /opt/gitlab/embedded/bin/bundle:25:in `<main>'
                Tasks: TOP => gitlab:db:configure
                (See full trace by running task with --trace)

      ================================================================================
      Error executing action `run` on resource 'bash_hide_env[migrate gitlab-rails database]'
      ================================================================================

      Mixlib::ShellOut::ShellCommandFailed
      ------------------------------------
      Expected process to exit with [0], but received '1'
      ---- Begin output of "bash"  ----
      STDOUT: rake aborted!
      ActiveRecord::ConnectionNotEstablished: could not connect to server: Connection refused
        Is the server running locally and accepting
        connections on Unix domain socket "/var/opt/gitlab/postgresql/.s.PGSQL.5432"?
      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:110:in `read_write'
      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:126:in `write_using_load_balancer'
      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:96:in `method_missing'
      /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:115:in `configure_database'
      /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:101:in `block (3 levels) in <top (required)>'
      /opt/gitlab/embedded/bin/bundle:25:in `load'
      /opt/gitlab/embedded/bin/bundle:25:in `<main>'

      Caused by:
      PG::ConnectionBad: could not connect to server: Connection refused
        Is the server running locally and accepting
        connections on Unix domain socket "/var/opt/gitlab/postgresql/.s.PGSQL.5432"?
      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:110:in `read_write'
      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:126:in `write_using_load_balancer'
      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:96:in `method_missing'
      /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:115:in `configure_database'
      /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:101:in `block (3 levels) in <top (required)>'
      /opt/gitlab/embedded/bin/bundle:25:in `load'
      /opt/gitlab/embedded/bin/bundle:25:in `<main>'
      Tasks: TOP => gitlab:db:configure
      (See full trace by running task with --trace)
      STDERR:
      ---- End output of "bash"  ----
      Ran "bash"  returned 1

      Cookbook Trace: (most recent call first)
      ----------------------------------------
      /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/resources/rails_migration.rb:20:in `block in class_from_file'

      Resource Declaration:
      ---------------------
      # In /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/resources/rails_migration.rb

       20:   bash_hide_env "migrate #{new_resource.name} database" do
       21:     code <<-EOH
       22:     set -e
       23:     log_file="#{logging_settings[:log_directory]}/#{new_resource.logfile_prefix}-$(date +%Y-%m-%d-%H-%M-%S).log"
       24:     umask 077
       25:     /opt/gitlab/bin/gitlab-rake #{new_resource.rake_task} 2>& 1 | tee ${log_file}
       26:     STATUS=${PIPESTATUS[0]}
       27:     chown #{account_helper.gitlab_user}:#{account_helper.gitlab_group} ${log_file}
       28:     echo $STATUS > #{new_resource.helper.db_migrate_status_file}
       29:     exit $STATUS
       30:     EOH
       31:
       32:     environment new_resource.environment if new_resource.property_is_set?(:environment)
       33:     new_resource.dependent_services.each do |svc|
       34:       notifies :restart, svc, :immediately
       35:     end
       36:
       37:     not_if { new_resource.helper.migrated? }
       38:     sensitive false
       39:   end
       40: end

      Compiled Resource:
      ------------------
      # Declared in /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/resources/rails_migration.rb:20:in `block in class_from_file'

      bash_hide_env("migrate gitlab-rails database") do
        action [:run]
        default_guard_interpreter :default
        interpreter "bash"
        declared_type :bash_hide_env
        cookbook_name "gitlab"
        recipe_name "database_migrations"
        code "    set -e\n    log_file=\"/var/log/gitlab/gitlab-rails/gitlab-rails-db-migrate-$(date +%Y-%m-%d-%H-%M-%S).log\"\n    umask 077\n    /opt/gitlab/bin/gitlab-rake gitlab:db:configure 2>& 1 | tee ${log_file}\n    STATUS=${PIPESTATUS[0]}\n    chown git:git ${log_file}\n    echo $STATUS > /var/opt/gitlab/gitlab-rails/upgrade-status/db-migrate-873248b1f0d3a7a5535771a3a1635803-917d3f7ad12\n    exit $STATUS\n"
        environment "*sensitive value suppressed*"
        sensitive false
        domain nil
        user nil
        not_if { #code block }
      end

      System Info:
      ------------
      chef_version=17.10.0
      platform=ubuntu
      platform_version=22.04
      ruby=ruby 3.0.6p216 (2023-03-30 revision 23a532679b) [x86_64-linux]
      program_name=/opt/gitlab/embedded/bin/cinc-client
      executable=/opt/gitlab/embedded/bin/cinc-client


    ================================================================================
    Error executing action `run` on resource 'rails_migration[gitlab-rails]'
    ================================================================================

    Mixlib::ShellOut::ShellCommandFailed
    ------------------------------------
    bash_hide_env[migrate gitlab-rails database] (gitlab::database_migrations line 20) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
    ---- Begin output of "bash"  ----
    STDOUT: rake aborted!
    ActiveRecord::ConnectionNotEstablished: could not connect to server: Connection refused
        Is the server running locally and accepting
        connections on Unix domain socket "/var/opt/gitlab/postgresql/.s.PGSQL.5432"?
    /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:110:in `read_write'
    /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:126:in `write_using_load_balancer'
    /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:96:in `method_missing'
    /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:115:in `configure_database'
    /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:101:in `block (3 levels) in <top (required)>'
    /opt/gitlab/embedded/bin/bundle:25:in `load'
    /opt/gitlab/embedded/bin/bundle:25:in `<main>'

    Caused by:
    PG::ConnectionBad: could not connect to server: Connection refused
        Is the server running locally and accepting
        connections on Unix domain socket "/var/opt/gitlab/postgresql/.s.PGSQL.5432"?
    /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:110:in `read_write'
    /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:126:in `write_using_load_balancer'
    /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:96:in `method_missing'
    /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:115:in `configure_database'
    /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:101:in `block (3 levels) in <top (required)>'
    /opt/gitlab/embedded/bin/bundle:25:in `load'
    /opt/gitlab/embedded/bin/bundle:25:in `<main>'
    Tasks: TOP => gitlab:db:configure
    (See full trace by running task with --trace)
    STDERR:
    ---- End output of "bash"  ----
    Ran "bash"  returned 1

    Cookbook Trace: (most recent call first)
    ----------------------------------------
    /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/resources/rails_migration.rb:20:in `block in class_from_file'

    Resource Declaration:
    ---------------------
    # In /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/database_migrations.rb

     51: rails_migration "gitlab-rails" do
     52:   rake_task 'gitlab:db:configure'
     53:   logfile_prefix 'gitlab-rails-db-migrate'
     54:   helper migration_helper
     55:
     56:   environment env_variables
     57:   dependent_services dependent_services
     58:   notifies :run, "execute[clear the gitlab-rails cache]", :immediately
     59:   notifies :run, "ruby_block[check remote PG version]", :immediately
     60:
     61:   only_if { migration_helper.attributes_node['auto_migrate'] }
     62: end

    Compiled Resource:
    ------------------
    # Declared in /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/database_migrations.rb:51:in `from_file'

    rails_migration("gitlab-rails") do
      action [:run]
      default_guard_interpreter :default
      declared_type :rails_migration
      cookbook_name "gitlab"
      recipe_name "database_migrations"
      rake_task "gitlab:db:configure"
      logfile_prefix "gitlab-rails-db-migrate"
      helper "*sensitive value suppressed*"
      environment "*sensitive value suppressed*"
      dependent_services []
      only_if { #code block }
    end

    System Info:
    ------------
    chef_version=17.10.0
    platform=ubuntu
    platform_version=22.04
    ruby=ruby 3.0.6p216 (2023-03-30 revision 23a532679b) [x86_64-linux]
    program_name=/opt/gitlab/embedded/bin/cinc-client
    executable=/opt/gitlab/embedded/bin/cinc-client

[2023-10-29T15:44:00+02:00] INFO: Running queued delayed notifications before re-raising exception
[2023-10-29T15:44:00+02:00] INFO: templatesymlink[Create a gitlab.yml and create a symlink to Rails root] sending run action to execute[clear the gitlab-rails cache] (delayed)
Recipe: gitlab::gitlab-rails
  * execute[clear the gitlab-rails cache] action run[2023-10-29T15:44:23+02:00] INFO: execute[clear the gitlab-rails cache] ran successfully

    - execute /opt/gitlab/bin/gitlab-rake cache:clear
[2023-10-29T15:44:23+02:00] INFO: version_file[Create version file for Gitaly] sending hup action to runit_service[gitaly] (delayed)
Recipe: gitaly::enable
  * runit_service[gitaly] action hup[2023-10-29T15:44:23+02:00] INFO: runit_service[gitaly] signalled (HUP)
[2023-10-29T15:44:23+02:00] INFO: runit_service[gitaly] sent hup

    - send hup to runit_service[gitaly]
[2023-10-29T15:44:23+02:00] INFO: version_file[Create version file for Gitlab KAS] sending restart action to runit_service[gitlab-kas] (delayed)
Recipe: gitlab-kas::enable
  * runit_service[gitlab-kas] action restart (up to date)

Running handlers:
[2023-10-29T15:44:33+02:00] ERROR: Running exception handlers
There was an error running gitlab-ctl reconfigure:

rails_migration[gitlab-rails] (gitlab::database_migrations line 51) had an error: Mixlib::ShellOut::ShellCommandFailed: bash_hide_env[migrate gitlab-rails database] (gitlab::database_migrations line 20) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
---- Begin output of "bash"  ----
STDOUT: rake aborted!
Running handlers complete
ActiveRecord::ConnectionNotEstablished: could not connect to server: Connection refused
        Is the server running locally and accepting
        connections on Unix domain socket "/var/opt/gitlab/postgresql/.s.PGSQL.5432"?
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:110:in `read_write'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:126:in `write_using_load_balancer'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:96:in `method_missing'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:115:in `configure_database'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:101:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:25:in `load'
/opt/gitlab/embedded/bin/bundle:25:in `<main>'
[2023-10-29T15:44:33+02:00] ERROR: Exception handlers complete

Caused by:
PG::ConnectionBad: could not connect to server: Connection refused
        Is the server running locally and accepting
        connections on Unix domain socket "/var/opt/gitlab/postgresql/.s.PGSQL.5432"?
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:110:in `read_write'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:126:in `write_using_load_balancer'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:96:in `method_missing'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:115:in `configure_database'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:101:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:25:in `load'
/opt/gitlab/embedded/bin/bundle:25:in `<main>'
Tasks: TOP => gitlab:db:configure
(See full trace by running task with --trace)
STDERR:
Infra Phase failed. 171 resources updated in 01 minutes 41 seconds
---- End output of "bash"  ----
Ran "bash"  returned 1

[2023-10-29T15:44:33+02:00] FATAL: Stacktrace dumped to /opt/gitlab/embedded/cookbooks/cache/cinc-stacktrace.out
[2023-10-29T15:44:33+02:00] FATAL: ---------------------------------------------------------------------------------------
[2023-10-29T15:44:33+02:00] FATAL: PLEASE PROVIDE THE CONTENTS OF THE stacktrace.out FILE (above) IF YOU FILE A BUG REPORT
[2023-10-29T15:44:33+02:00] FATAL: ---------------------------------------------------------------------------------------
[2023-10-29T15:44:33+02:00] FATAL: Mixlib::ShellOut::ShellCommandFailed: rails_migration[gitlab-rails] (gitlab::database_migrations line 51) had an error: Mixlib::ShellOut::ShellCommandFailed: bash_hide_env[migrate gitlab-rails database] (gitlab::database_migrations line 20) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
---- Begin output of "bash"  ----
STDOUT: rake aborted!
ActiveRecord::ConnectionNotEstablished: could not connect to server: Connection refused
        Is the server running locally and accepting
        connections on Unix domain socket "/var/opt/gitlab/postgresql/.s.PGSQL.5432"?
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:110:in `read_write'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:126:in `write_using_load_balancer'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:96:in `method_missing'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:115:in `configure_database'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:101:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:25:in `load'
/opt/gitlab/embedded/bin/bundle:25:in `<main>'

Caused by:
PG::ConnectionBad: could not connect to server: Connection refused
        Is the server running locally and accepting
        connections on Unix domain socket "/var/opt/gitlab/postgresql/.s.PGSQL.5432"?
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:110:in `read_write'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:126:in `write_using_load_balancer'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:96:in `method_missing'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:115:in `configure_database'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:101:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:25:in `load'
/opt/gitlab/embedded/bin/bundle:25:in `<main>'
Tasks: TOP => gitlab:db:configure
(See full trace by running task with --trace)
STDERR:
---- End output of "bash"  ----
Ran "bash"  returned 1

I’ve tried:

  • Upgrading directly to later versions (16.2.0, 16.3.0)
  • Restoring from a backup and trying to upgrade
  • Upgrading from a different host
  • Trying the 15.9 workaround, the 15.10 workaround and the 15.11 workaround

I’m not sure what I’m missing…

Any guidance would be greatly appreciated!

2

Did you stop Gitlab when attempting to upgrade? Usually this is because the services were stopped. You are meant to leave Gitlab running, the upgrade process will automatically stop the appropriate services and restart afterwards.

For DB migrations to work, Postgres needs to be running, this is why it failed.

3

Thanks for the quick response! I’ve run the workaround on the live 15.11.13 instance, pulled the new 16 image, and brought the instance up (I’m using docker compose, so docker compose up -d). This error is during the initialization of the container - and the startup fails after it.

4

Hello, I have same problems ! @MH6 du you have found a solution ?

5

I have the same problem. I use docker-compose, the following is my upgrade path, everything is normal before version 16

# image: gitlab/gitlab-ce:14.3.3-ce.0
# image: gitlab/gitlab-ce:14.3.6-ce.0
# image: gitlab/gitlab-ce:14.9.5-ce.0
# image: gitlab/gitlab-ce:14.10.5-ce.0
# image: gitlab/gitlab-ce:15.0.5-ce.0
# image: gitlab/gitlab-ce:15.4.6-ce.0
# image: gitlab/gitlab-ce:15.11.13-ce.0
image: gitlab/gitlab-ce:16.0.2-ce.0
6

The 16.0.2 version of psql has not been upgraded successfully. It needs to be manually upgraded to version 13.