Upgrade from 10.2.4-ce.0 to 10.2.5-ce.0 fails, 502 error

Hi,
Wondering what the best way around this upgrade error is:

Ubuntu, “16.04.3 LTS (Xenial Xerus)” fails with errors:
Error executing action run on resource ‘bash[migrate gitlab-rails database]’
Error executing action run on resource ‘execute[clear the gitlab-rails cache]’
Quite a simple omnibus install, with LDAP and mattermost enabled, very few projects.
Two bundle processes using 100% CPU. and:
502
Whoops, GitLab is taking too much time to respond
When going to the login page.
Reverted to snapshot of before upgrade and tried the upgrade a few times with identical results.

Some of the upgrade log, the bits containing the errors below:

  • execute[/opt/gitlab/embedded/bin/initdb -D /var/opt/gitlab/postgresql/data -E UTF8] action run (skipped due to not_if)

  • template[/var/opt/gitlab/postgresql/data/postgresql.conf] action create (up to date)

  • template[/var/opt/gitlab/postgresql/data/runtime.conf] action create (up to date)

  • template[/var/opt/gitlab/postgresql/data/pg_hba.conf] action create (up to date)

  • template[/var/opt/gitlab/postgresql/data/pg_ident.conf] action create (up to date)

  • directory[/opt/gitlab/sv/postgresql] action create (up to date)

  • directory[/opt/gitlab/sv/postgresql/log] action create (up to date)

  • directory[/opt/gitlab/sv/postgresql/log/main] action create (up to date)

  • template[/opt/gitlab/sv/postgresql/run] action create (up to date)

  • template[/opt/gitlab/sv/postgresql/log/run] action create (up to date)

  • template[/var/log/gitlab/postgresql/config] action create (up to date)

  • ruby_block[reload postgresql svlogd configuration] action nothing (skipped due to action :nothing)

  • file[/opt/gitlab/sv/postgresql/down] action delete (up to date)

  • directory[/opt/gitlab/sv/postgresql/control] action create (up to date)

  • template[/opt/gitlab/sv/postgresql/control/t] action create (up to date)

  • link[/opt/gitlab/init/postgresql] action create (up to date)

  • link[/opt/gitlab/service/postgresql] action create (up to date)

  • ruby_block[supervise_postgresql_sleep] action run (skipped due to not_if)

  • directory[/opt/gitlab/sv/postgresql/supervise] action create (up to date)

  • directory[/opt/gitlab/sv/postgresql/log/supervise] action create (up to date)

  • file[/opt/gitlab/sv/postgresql/supervise/ok] action touch (skipped due to only_if)

  • file[/opt/gitlab/sv/postgresql/log/supervise/ok] action touch (skipped due to only_if)

  • file[/opt/gitlab/sv/postgresql/supervise/control] action touch (skipped due to only_if)

  • file[/opt/gitlab/sv/postgresql/log/supervise/control] action touch (skipped due to only_if)

  • service[postgresql] action nothing (skipped due to action :nothing)
    Recipe: gitlab::postgresql-bin

  • ruby_block[Link postgresql bin files to the correct version] action run (skipped due to only_if)
    Recipe: gitlab::postgresql

  • template[/opt/gitlab/etc/gitlab-psql-rc] action create (up to date)

  • postgresql_user[gitlab] action create

    • execute[create gitlab postgresql user] action run (skipped due to not_if)
    • execute[set password for gitlab postgresql user] action run (skipped due to not_if)
      (up to date)
  • execute[create gitlabhq_production database] action run (skipped due to not_if)

  • postgresql_user[gitlab_replicator] action create

    • execute[create gitlab_replicator postgresql user] action run (skipped due to not_if)
    • execute[set password for gitlab_replicator postgresql user] action run (skipped due to not_if)
      (up to date)
  • execute[enable pg_trgm extension] action run (skipped due to not_if)

  • execute[reload postgresql] action nothing (skipped due to action :nothing)

  • execute[start postgresql] action nothing (skipped due to action :nothing)
    Recipe: gitlab::database_migrations

  • bash[migrate gitlab-rails database] action run
    [execute] rake aborted!
    NoMethodError: undefined method each' for nil:NilClass /opt/gitlab/embedded/service/gitlab-rails/config/initializers/1_settings.rb:143:in <top (required)>’
    /opt/gitlab/embedded/service/gitlab-rails/config/environment.rb:5:in <top (required)>' /opt/gitlab/embedded/bin/bundle:23:in load’
    /opt/gitlab/embedded/bin/bundle:23:in `’
    Tasks: TOP => gitlab:db:configure => environment
    (See full trace by running task with --trace)

    ================================================================================
    Error executing action run on resource ‘bash[migrate gitlab-rails database]’

    Mixlib::ShellOut::ShellCommandFailed

    Expected process to exit with [0], but received ‘1’
    ---- Begin output of “bash” “/tmp/chef-script20171220-11678-sdiy7b” ----
    STDOUT: rake aborted!
    NoMethodError: undefined method each' for nil:NilClass /opt/gitlab/embedded/service/gitlab-rails/config/initializers/1_settings.rb:143:in <top (required)>’
    /opt/gitlab/embedded/service/gitlab-rails/config/environment.rb:5:in <top (required)>' /opt/gitlab/embedded/bin/bundle:23:in load’
    /opt/gitlab/embedded/bin/bundle:23:in `’
    Tasks: TOP => gitlab:db:configure => environment
    (See full trace by running task with --trace)
    STDERR:
    ---- End output of “bash” “/tmp/chef-script20171220-11678-sdiy7b” ----
    Ran “bash” “/tmp/chef-script20171220-11678-sdiy7b” returned 1

    Resource Declaration:

    In /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/database_migrations.rb

    51: bash “migrate gitlab-rails database” do
    52: code <<-EOH
    53: set -e
    54: log_file="#{node[‘gitlab’][‘gitlab-rails’][‘log_directory’]}/gitlab-rails-db-migrate-(date +%Y-%m-%d-%H-%M-%S).log" 55: umask 077 56: /opt/gitlab/bin/gitlab-rake gitlab:db:configure 2>& 1 | tee {log_file}
    57: STATUS=${PIPESTATUS[0]}
    58: echo $STATUS > #{db_migrate_status_file}
    59: exit $STATUS
    60: EOH
    61: environment env_variables unless env_variables.empty?
    62: notifies :run, “execute[clear the gitlab-rails cache]”, :immediately
    63: dependent_services.each do |svc|
    64: notifies :restart, svc, :immediately
    65: end
    66: not_if “(test -f #{db_migrate_status_file}) && (cat #{db_migrate_status_file} | grep -Fx 0)”
    67: only_if { node[‘gitlab’][‘gitlab-rails’][‘auto_migrate’] }
    68: end

    Compiled Resource:

    Declared in /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/database_migrations.rb:51:in `from_file’

    bash(“migrate gitlab-rails database”) do
    action [:run]
    retries 0
    retry_delay 2
    default_guard_interpreter :default
    command “migrate gitlab-rails database”
    backup 5
    returns 0
    code " set -e\n log_file="/var/log/gitlab/gitlab-rails/gitlab-rails-db-migrate-(date +%Y-%m-%d-%H-%M-%S).log\"\n umask 077\n /opt/gitlab/bin/gitlab-rake gitlab:db:configure 2>& 1 | tee {log_file}\n STATUS=${PIPESTATUS[0]}\n echo $STATUS > /var/opt/gitlab/gitlab-rails/upgrade-status/db-migrate-873248b1f0d3a7a5535771a3a1635803-16502c8\n exit $STATUS\n"
    interpreter “bash”
    declared_type :bash
    cookbook_name “gitlab”
    recipe_name “database_migrations”
    not_if “(test -f /var/opt/gitlab/gitlab-rails/upgrade-status/db-migrate-873248b1f0d3a7a5535771a3a1635803-16502c8) && (cat /var/opt/gitlab/gitlab-rails/upgrade-status/db-migrate-873248b1f0d3a7a5535771a3a1635803-16502c8 | grep -Fx 0)”
    only_if { #code block }
    end

    Platform:

    x86_64-linux

Recipe: gitlab::gitlab-rails

  • execute[clear the gitlab-rails cache] action run
    [execute] rake aborted!
    NoMethodError: undefined method each' for nil:NilClass /opt/gitlab/embedded/service/gitlab-rails/config/initializers/1_settings.rb:143:in <top (required)>’
    /opt/gitlab/embedded/service/gitlab-rails/config/environment.rb:5:in <top (required)>' /opt/gitlab/embedded/bin/bundle:23:in load’
    /opt/gitlab/embedded/bin/bundle:23:in `’
    Tasks: TOP => cache:clear => cache:clear:redis => environment
    (See full trace by running task with --trace)

    ================================================================================
    Error executing action run on resource ‘execute[clear the gitlab-rails cache]’

    Mixlib::ShellOut::ShellCommandFailed

    Expected process to exit with [0], but received ‘1’
    ---- Begin output of /opt/gitlab/bin/gitlab-rake cache:clear ----
    STDOUT:
    STDERR: rake aborted!
    NoMethodError: undefined method each' for nil:NilClass /opt/gitlab/embedded/service/gitlab-rails/config/initializers/1_settings.rb:143:in <top (required)>’
    /opt/gitlab/embedded/service/gitlab-rails/config/environment.rb:5:in <top (required)>' /opt/gitlab/embedded/bin/bundle:23:in load’
    /opt/gitlab/embedded/bin/bundle:23:in `’
    Tasks: TOP => cache:clear => cache:clear:redis => environment
    (See full trace by running task with --trace)
    ---- End output of /opt/gitlab/bin/gitlab-rake cache:clear ----
    Ran /opt/gitlab/bin/gitlab-rake cache:clear returned 1

    Resource Declaration:

    In /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/gitlab-rails.rb

    367: execute “clear the gitlab-rails cache” do
    368: command “/opt/gitlab/bin/gitlab-rake cache:clear”
    369: action :nothing
    370: not_if { omnibus_helper.not_listening?(‘redis’) || !node[‘gitlab’][‘gitlab-rails’][‘rake_cache_clear’] }
    371: end
    372:

    Compiled Resource:

    Declared in /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/gitlab-rails.rb:367:in `from_file’

    execute(“clear the gitlab-rails cache”) do
    action [:nothing]
    retries 0
    retry_delay 2
    default_guard_interpreter :execute
    command “/opt/gitlab/bin/gitlab-rake cache:clear”
    backup 5
    returns 0
    declared_type :execute
    cookbook_name “gitlab”
    recipe_name “gitlab-rails”
    not_if { #code block }
    end

    Platform:

    x86_64-linux

Running handlers:
Running handlers complete
Chef Client failed. 8 resources updated in 49 seconds
Checking for an omnibus managed postgresql: OK
Checking for a newer version of PostgreSQL to install
No new version of PostgreSQL installed, nothing to upgrade to
Ensuring PostgreSQL is updated: OK
Restarting previously running GitLab services
ok: run: gitaly: (pid 12243) 0s
ok: run: gitlab-monitor: (pid 12253) 0s
ok: run: gitlab-workhorse: (pid 12257) 1s
ok: run: logrotate: (pid 12269) 0s
ok: run: mattermost: (pid 12275) 1s
ok: run: nginx: (pid 12283) 0s
ok: run: node-exporter: (pid 12291) 1s
ok: run: postgres-exporter: (pid 12301) 0s
ok: run: postgresql: (pid 11657) 57s
ok: run: prometheus: (pid 12309) 0s
ok: run: redis: (pid 11665) 57s
ok: run: redis-exporter: (pid 12318) 1s
ok: run: sidekiq: (pid 12327) 0s
ok: run: unicorn: (pid 12334) 1s

 _______ __  __          __
/ ____(_) /_/ /   ____ _/ /_

/ / / / / / / __ `/ __
/ // / / // /
/ // / // /
_
//_/___/_,/./

Upgrade complete! If your GitLab server is misbehaving try running
sudo gitlab-ctl restart
before anything else.
If you need to roll back to the previous version you can use the database
backup made during the upgrade (scroll up for the filename).

csgadmin@gitlab:~$ sudo gitlab-ctl restart
ok: run: gitaly: (pid 12762) 0s
ok: run: gitlab-monitor: (pid 12779) 1s
ok: run: gitlab-workhorse: (pid 12783) 0s
ok: run: logrotate: (pid 12792) 1s
ok: run: mattermost: (pid 12806) 0s
ok: run: nginx: (pid 12812) 0s
ok: run: node-exporter: (pid 12824) 1s
ok: run: postgres-exporter: (pid 12831) 0s
ok: run: postgresql: (pid 12840) 1s
ok: run: prometheus: (pid 12848) 0s
ok: run: redis: (pid 12855) 1s
ok: run: redis-exporter: (pid 12860) 0s
ok: run: sidekiq: (pid 12868) 0s
ok: run: unicorn: (pid 12877) 1s