Upgrade failed, now stuck with 502

I tried to upgrade my gitlab-ce server on debain to the latest version with apt-get upgrade. I always keep my server up-to-date and normally it just updates, but this time the upgrade failed. After that my gitlab server does not provide access to users anymore. It fails with 502 error taking too much time to respond.

Log of the upgrade:
** Invoke gitlab:backup:create (first_time)
** Invoke gitlab_environment (first_time)
** Invoke environment (first_time)
** Execute environment
rake aborted!
Psych::SyntaxError: (): mapping values are not allowed in this context at line 44 column 40
/opt/gitlab/embedded/lib/ruby/2.3.0/psych.rb:379:in parse' /opt/gitlab/embedded/lib/ruby/2.3.0/psych.rb:379:in parse_stream’
/opt/gitlab/embedded/lib/ruby/2.3.0/psych.rb:327:in parse' /opt/gitlab/embedded/lib/ruby/2.3.0/psych.rb:254:in load’
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/settingslogic-2.0.9/lib/settingslogic.rb:103:in initialize' /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/settingslogic-2.0.9/lib/settingslogic.rb:60:in new’
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/settingslogic-2.0.9/lib/settingslogic.rb:60:in instance' /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/settingslogic-2.0.9/lib/settingslogic.rb:37:in ’
/opt/gitlab/embedded/service/gitlab-rails/config/initializers/1_settings.rb:131:in <top (required)>' /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activesupport-4.2.10/lib/active_support/dependencies.rb:268:in load’
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activesupport-4.2.10/lib/active_support/dependencies.rb:268:in block in load' /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activesupport-4.2.10/lib/active_support/dependencies.rb:240:in load_dependency’
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activesupport-4.2.10/lib/active_support/dependencies.rb:268:in load' /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/railties-4.2.10/lib/rails/engine.rb:652:in block in load_config_initializer’
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activesupport-4.2.10/lib/active_support/notifications.rb:166:in instrument' /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/railties-4.2.10/lib/rails/engine.rb:651:in load_config_initializer’
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/railties-4.2.10/lib/rails/engine.rb:616:in block (2 levels) in <class:Engine>' /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/railties-4.2.10/lib/rails/engine.rb:615:in each’
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/railties-4.2.10/lib/rails/engine.rb:615:in block in <class:Engine>' /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/railties-4.2.10/lib/rails/initializable.rb:30:in instance_exec’
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/railties-4.2.10/lib/rails/initializable.rb:30:in run' /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/railties-4.2.10/lib/rails/initializable.rb:55:in block in run_initializers’
/opt/gitlab/embedded/lib/ruby/2.3.0/tsort.rb:228:in block in tsort_each' /opt/gitlab/embedded/lib/ruby/2.3.0/tsort.rb:350:in block (2 levels) in each_strongly_connected_component’
/opt/gitlab/embedded/lib/ruby/2.3.0/tsort.rb:422:in block (2 levels) in each_strongly_connected_component_from' /opt/gitlab/embedded/lib/ruby/2.3.0/tsort.rb:431:in each_strongly_connected_component_from’
/opt/gitlab/embedded/lib/ruby/2.3.0/tsort.rb:421:in block in each_strongly_connected_component_from' /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/railties-4.2.10/lib/rails/initializable.rb:44:in each’
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/railties-4.2.10/lib/rails/initializable.rb:44:in tsort_each_child' /opt/gitlab/embedded/lib/ruby/2.3.0/tsort.rb:415:in call’
/opt/gitlab/embedded/lib/ruby/2.3.0/tsort.rb:415:in each_strongly_connected_component_from' /opt/gitlab/embedded/lib/ruby/2.3.0/tsort.rb:349:in block in each_strongly_connected_component’
/opt/gitlab/embedded/lib/ruby/2.3.0/tsort.rb:347:in each' /opt/gitlab/embedded/lib/ruby/2.3.0/tsort.rb:347:in call’
/opt/gitlab/embedded/lib/ruby/2.3.0/tsort.rb:347:in each_strongly_connected_component' /opt/gitlab/embedded/lib/ruby/2.3.0/tsort.rb:226:in tsort_each’
/opt/gitlab/embedded/lib/ruby/2.3.0/tsort.rb:205:in tsort_each' /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/railties-4.2.10/lib/rails/initializable.rb:54:in run_initializers’
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/railties-4.2.10/lib/rails/application.rb:352:in initialize!' /opt/gitlab/embedded/service/gitlab-rails/config/environment.rb:5:in <top (required)>’
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activesupport-4.2.10/lib/active_support/dependencies.rb:274:in require' /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activesupport-4.2.10/lib/active_support/dependencies.rb:274:in block in require’
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activesupport-4.2.10/lib/active_support/dependencies.rb:240:in load_dependency' /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activesupport-4.2.10/lib/active_support/dependencies.rb:274:in require’
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/railties-4.2.10/lib/rails/application.rb:328:in require_environment!' /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/railties-4.2.10/lib/rails/application.rb:457:in block in run_tasks_blocks’
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/rake-12.3.0/lib/rake/task.rb:251:in block in execute' /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/rake-12.3.0/lib/rake/task.rb:251:in each’
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/rake-12.3.0/lib/rake/task.rb:251:in execute' /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/rake-12.3.0/lib/rake/task.rb:195:in block in invoke_with_call_chain’
/opt/gitlab/embedded/lib/ruby/2.3.0/monitor.rb:214:in mon_synchronize' /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/rake-12.3.0/lib/rake/task.rb:188:in invoke_with_call_chain’
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/rake-12.3.0/lib/rake/task.rb:217:in block in invoke_prerequisites' /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/rake-12.3.0/lib/rake/task.rb:215:in each’
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/rake-12.3.0/lib/rake/task.rb:215:in invoke_prerequisites' /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/rake-12.3.0/lib/rake/task.rb:194:in block in invoke_with_call_chain’
/opt/gitlab/embedded/lib/ruby/2.3.0/monitor.rb:214:in mon_synchronize' /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/rake-12.3.0/lib/rake/task.rb:188:in invoke_with_call_chain’
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/rake-12.3.0/lib/rake/task.rb:217:in block in invoke_prerequisites' /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/rake-12.3.0/lib/rake/task.rb:215:in each’
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/rake-12.3.0/lib/rake/task.rb:215:in invoke_prerequisites' /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/rake-12.3.0/lib/rake/task.rb:194:in block in invoke_with_call_chain’
/opt/gitlab/embedded/lib/ruby/2.3.0/monitor.rb:214:in mon_synchronize' /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/rake-12.3.0/lib/rake/task.rb:188:in invoke_with_call_chain’
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/rake-12.3.0/lib/rake/task.rb:181:in invoke' /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/rake-12.3.0/lib/rake/application.rb:160:in invoke_task’
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/rake-12.3.0/lib/rake/application.rb:116:in block (2 levels) in top_level' /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/rake-12.3.0/lib/rake/application.rb:116:in each’
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/rake-12.3.0/lib/rake/application.rb:116:in block in top_level' /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/rake-12.3.0/lib/rake/application.rb:125:in run_with_threads’
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/rake-12.3.0/lib/rake/application.rb:110:in top_level' /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/rake-12.3.0/lib/rake/application.rb:83:in block in run’
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/rake-12.3.0/lib/rake/application.rb:186:in standard_exception_handling' /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/rake-12.3.0/lib/rake/application.rb:80:in run’
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/rake-12.3.0/exe/rake:27:in <top (required)>' /opt/gitlab/embedded/bin/rake:23:in load’
/opt/gitlab/embedded/bin/rake:23:in <top (required)>' /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/bundler-1.13.7/lib/bundler/cli/exec.rb:74:in load’
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/bundler-1.13.7/lib/bundler/cli/exec.rb:74:in kernel_load' /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/bundler-1.13.7/lib/bundler/cli/exec.rb:27:in run’
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/bundler-1.13.7/lib/bundler/cli.rb:332:in exec' /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/bundler-1.13.7/lib/bundler/vendor/thor/lib/thor/command.rb:27:in run’
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/bundler-1.13.7/lib/bundler/vendor/thor/lib/thor/invocation.rb:126:in invoke_command' /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/bundler-1.13.7/lib/bundler/vendor/thor/lib/thor.rb:359:in dispatch’
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/bundler-1.13.7/lib/bundler/cli.rb:20:in dispatch' /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/bundler-1.13.7/lib/bundler/vendor/thor/lib/thor/base.rb:440:in start’
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/bundler-1.13.7/lib/bundler/cli.rb:11:in start' /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/bundler-1.13.7/exe/bundle:34:in block in <top (required)>’
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/bundler-1.13.7/lib/bundler/friendly_errors.rb:100:in with_friendly_errors' /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/bundler-1.13.7/exe/bundle:26:in <top (required)>’
/opt/gitlab/embedded/bin/bundle:23:in load' /opt/gitlab/embedded/bin/bundle:23:in ’
Tasks: TOP => gitlab:backup:create => gitlab_environment => environment

Doing research on the issue, I can’t find anything in relation to Gitlab. I just did the upgrade via the package archive, why did it went wrong? and how to fix the server so that devs can start using it again?

Thank you for any help!

I disabled the backup of the database and was able to execute apt-get upgrade. I moved from 10.6.2 to 10.6.4, however I’m still on 502. So I ran a reconfigure and the output contains an error:

Recipe: gitlab::database_migrationse[0m

• bash[migrate gitlab-rails database] action run
  [execute] rake aborted!
  Psych::SyntaxError: (): mapping values are not allowed in this context at line 44 column 40
  /opt/gitlab/embedded/service/gitlab-rails/config/initializers/1_settings.rb:131:in <top (required)>' /opt/gitlab/embedded/service/gitlab-rails/config/environment.rb:5:in <top (required)>’
  /opt/gitlab/embedded/bin/bundle:23:in load' /opt/gitlab/embedded/bin/bundle:23:in ’
  Tasks: TOP => gitlab:db:configure => environment
  (See full trace by running task with --trace)
  e[0m
  ================================================================================e[0m
  e[31mError executing action run on resource 'bash[migrate gitlab-rails database]'e[0m
  ================================================================================e[0m

e[0m Mixlib::ShellOut::ShellCommandFailede[0m
------------------------------------e[0m
Expected process to exit with [0], but received ‘1’
e[0m ---- Begin output of “bash” “/tmp/chef-script20180420-3449-nbbm4m” ----
e[0m STDOUT: rake aborted!
e[0m Psych::SyntaxError: (): mapping values are not allowed in this context at line 44 column 40
e[0m /opt/gitlab/embedded/service/gitlab-rails/config/initializers/1_settings.rb:131:in <top (required)>' e[0m /opt/gitlab/embedded/service/gitlab-rails/config/environment.rb:5:in <top (required)>’
e[0m /opt/gitlab/embedded/bin/bundle:23:in load' e[0m /opt/gitlab/embedded/bin/bundle:23:in ’
e[0m Tasks: TOP => gitlab:db:configure => environment
e[0m (See full trace by running task with --trace)
e[0m STDERR:
e[0m ---- End output of “bash” “/tmp/chef-script20180420-3449-nbbm4m” ----
e[0m Ran “bash” “/tmp/chef-script20180420-3449-nbbm4m” returned 1e[0m

e[0m Resource Declaration:e[0m
---------------------e[0m
# In /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/database_migrations.rb
e[0m
e[0m 49: bash “migrate gitlab-rails database” do
e[0m 50: code <<-EOH
e[0m 51: set -e
e[0m 52: log_file="#{node[‘gitlab’][‘gitlab-rails’][‘log_directory’]}/gitlab-rails-db-migrate-(date +%Y-%m-%d-%H-%M-%S).log" e[0m 53: umask 077 e[0m 54: /opt/gitlab/bin/gitlab-rake gitlab:db:configure 2>& 1 | tee {log_file}
e[0m 55: STATUS=${PIPESTATUS[0]}
e[0m 56: echo $STATUS > #{db_migrate_status_file}
e[0m 57: exit STATUS e[0m 58: EOH e[0m 59: environment env_variables unless env_variables.empty? e[0m 60: notifies :run, "execute[clear the gitlab-rails cache]", :immediately e[0m 61: dependent_services.each do |svc| e[0m 62: notifies :restart, svc, :immediately e[0m 63: end e[0m 64: not_if "(test -f #{db_migrate_status_file}) && (cat #{db_migrate_status_file} | grep -Fx 0)" e[0m 65: only_if { node['gitlab']['gitlab-rails']['auto_migrate'] } e[0m 66: end e[0m e[0m Compiled Resource:e[0m ------------------e[0m # Declared in /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/database_migrations.rb:49:in `from_file' e[0m e[0m bash("migrate gitlab-rails database") do e[0m action [:run] e[0m default_guard_interpreter :default e[0m command nil e[0m backup 5 e[0m returns 0 e[0m user nil e[0m interpreter "bash" e[0m declared_type :bash e[0m cookbook_name "gitlab" e[0m recipe_name "database_migrations" e[0m code " set -e\n log_file=\"/var/log/gitlab/gitlab-rails/gitlab-rails-db-migrate-(date +%Y-%m-%d-%H-%M-%S).log"\n umask 077\n /opt/gitlab/bin/gitlab-rake gitlab:db:configure 2>& 1 | tee {log_file}\n STATUS={PIPESTATUS[0]}\n echo $STATUS > /var/opt/gitlab/gitlab-rails/upgrade-status/db-migrate-873248b1f0d3a7a5535771a3a1635803-dee2c87\n exit $STATUS\n"
e[0m domain nil
e[0m not_if “(test -f /var/opt/gitlab/gitlab-rails/upgrade-status/db-migrate-873248b1f0d3a7a5535771a3a1635803-dee2c87) && (cat /var/opt/gitlab/gitlab-rails/upgrade-status/db-migrate-873248b1f0d3a7a5535771a3a1635803-dee2c87 | grep -Fx 0)”
e[0m only_if { #code block }
e[0m end
e[0m
e[0m System Info:e[0m
------------e[0m
chef_version=13.6.4
e[0m platform=debian
e[0m platform_version=8.10
e[0m ruby=ruby 2.3.6p384 (2017-12-14 revision 61254) [x86_64-linux]
e[0m program_name=/opt/gitlab/embedded/bin/chef-client
e[0m executable=/opt/gitlab/embedded/bin/chef-cliente[0m

e[0me[0m
Running handlers:e[0m
There was an error running gitlab-ctl reconfigure:

bash[migrate gitlab-rails database] (gitlab::database_migrations line 49) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received ‘1’
---- Begin output of “bash” “/tmp/chef-script20180420-3449-nbbm4m” ----
STDOUT: rake aborted!
Psych::SyntaxError: (): mapping values are not allowed in this context at line 44 column 40
/opt/gitlab/embedded/service/gitlab-rails/config/initializers/1_settings.rb:131:in <top (required)>' /opt/gitlab/embedded/service/gitlab-rails/config/environment.rb:5:in <top (required)>’
/opt/gitlab/embedded/bin/bundle:23:in load' /opt/gitlab/embedded/bin/bundle:23:in ’
Tasks: TOP => gitlab:db:configure => environment
(See full trace by running task with --trace)
STDERR:
---- End output of “bash” “/tmp/chef-script20180420-3449-nbbm4m” ----
Ran “bash” “/tmp/chef-script20180420-3449-nbbm4m” returned 1

Running handlers complete
e[0mChef Client failed. 2 resources updated in 02 minutes 39 secondse[0m

I’m still troubleshooting this. Any pointers/help is appreciated!