Hi,
Wondering what the best way around this upgrade error is:
Ubuntu, “16.04.3 LTS (Xenial Xerus)” fails with errors:
Error executing action run
on resource ‘bash[migrate gitlab-rails database]’
Error executing action run
on resource ‘execute[clear the gitlab-rails cache]’
Quite a simple omnibus install, with LDAP and mattermost enabled, very few projects.
Two bundle processes using 100% CPU. and:
502
Whoops, GitLab is taking too much time to respond
When going to the login page.
Reverted to snapshot of before upgrade and tried the upgrade a few times with identical results.
Some of the upgrade log, the bits containing the errors below:
-
execute[/opt/gitlab/embedded/bin/initdb -D /var/opt/gitlab/postgresql/data -E UTF8] action run (skipped due to not_if)
-
template[/var/opt/gitlab/postgresql/data/postgresql.conf] action create (up to date)
-
template[/var/opt/gitlab/postgresql/data/runtime.conf] action create (up to date)
-
template[/var/opt/gitlab/postgresql/data/pg_hba.conf] action create (up to date)
-
template[/var/opt/gitlab/postgresql/data/pg_ident.conf] action create (up to date)
-
directory[/opt/gitlab/sv/postgresql] action create (up to date)
-
directory[/opt/gitlab/sv/postgresql/log] action create (up to date)
-
directory[/opt/gitlab/sv/postgresql/log/main] action create (up to date)
-
template[/opt/gitlab/sv/postgresql/run] action create (up to date)
-
template[/opt/gitlab/sv/postgresql/log/run] action create (up to date)
-
template[/var/log/gitlab/postgresql/config] action create (up to date)
-
ruby_block[reload postgresql svlogd configuration] action nothing (skipped due to action :nothing)
-
file[/opt/gitlab/sv/postgresql/down] action delete (up to date)
-
directory[/opt/gitlab/sv/postgresql/control] action create (up to date)
-
template[/opt/gitlab/sv/postgresql/control/t] action create (up to date)
-
link[/opt/gitlab/init/postgresql] action create (up to date)
-
link[/opt/gitlab/service/postgresql] action create (up to date)
-
ruby_block[supervise_postgresql_sleep] action run (skipped due to not_if)
-
directory[/opt/gitlab/sv/postgresql/supervise] action create (up to date)
-
directory[/opt/gitlab/sv/postgresql/log/supervise] action create (up to date)
-
file[/opt/gitlab/sv/postgresql/supervise/ok] action touch (skipped due to only_if)
-
file[/opt/gitlab/sv/postgresql/log/supervise/ok] action touch (skipped due to only_if)
-
file[/opt/gitlab/sv/postgresql/supervise/control] action touch (skipped due to only_if)
-
file[/opt/gitlab/sv/postgresql/log/supervise/control] action touch (skipped due to only_if)
-
service[postgresql] action nothing (skipped due to action :nothing)
Recipe: gitlab::postgresql-bin -
ruby_block[Link postgresql bin files to the correct version] action run (skipped due to only_if)
Recipe: gitlab::postgresql -
template[/opt/gitlab/etc/gitlab-psql-rc] action create (up to date)
-
postgresql_user[gitlab] action create
- execute[create gitlab postgresql user] action run (skipped due to not_if)
- execute[set password for gitlab postgresql user] action run (skipped due to not_if)
(up to date)
-
execute[create gitlabhq_production database] action run (skipped due to not_if)
-
postgresql_user[gitlab_replicator] action create
- execute[create gitlab_replicator postgresql user] action run (skipped due to not_if)
- execute[set password for gitlab_replicator postgresql user] action run (skipped due to not_if)
(up to date)
-
execute[enable pg_trgm extension] action run (skipped due to not_if)
-
execute[reload postgresql] action nothing (skipped due to action :nothing)
-
execute[start postgresql] action nothing (skipped due to action :nothing)
Recipe: gitlab::database_migrations -
bash[migrate gitlab-rails database] action run
[execute] rake aborted!
NoMethodError: undefined methodeach' for nil:NilClass /opt/gitlab/embedded/service/gitlab-rails/config/initializers/1_settings.rb:143:in
<top (required)>’
/opt/gitlab/embedded/service/gitlab-rails/config/environment.rb:5:in<top (required)>' /opt/gitlab/embedded/bin/bundle:23:in
load’
/opt/gitlab/embedded/bin/bundle:23:in `’
Tasks: TOP => gitlab:db:configure => environment
(See full trace by running task with --trace)================================================================================
Error executing actionrun
on resource ‘bash[migrate gitlab-rails database]’Mixlib::ShellOut::ShellCommandFailed
Expected process to exit with [0], but received ‘1’
---- Begin output of “bash” “/tmp/chef-script20171220-11678-sdiy7b” ----
STDOUT: rake aborted!
NoMethodError: undefined methodeach' for nil:NilClass /opt/gitlab/embedded/service/gitlab-rails/config/initializers/1_settings.rb:143:in
<top (required)>’
/opt/gitlab/embedded/service/gitlab-rails/config/environment.rb:5:in<top (required)>' /opt/gitlab/embedded/bin/bundle:23:in
load’
/opt/gitlab/embedded/bin/bundle:23:in `’
Tasks: TOP => gitlab:db:configure => environment
(See full trace by running task with --trace)
STDERR:
---- End output of “bash” “/tmp/chef-script20171220-11678-sdiy7b” ----
Ran “bash” “/tmp/chef-script20171220-11678-sdiy7b” returned 1Resource Declaration:
In /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/database_migrations.rb
51: bash “migrate gitlab-rails database” do
52: code <<-EOH
53: set -e
54: log_file="#{node[‘gitlab’][‘gitlab-rails’][‘log_directory’]}/gitlab-rails-db-migrate-(date +%Y-%m-%d-%H-%M-%S).log" 55: umask 077 56: /opt/gitlab/bin/gitlab-rake gitlab:db:configure 2>& 1 | tee {log_file}
57: STATUS=${PIPESTATUS[0]}
58: echo $STATUS > #{db_migrate_status_file}
59: exit $STATUS
60: EOH
61: environment env_variables unless env_variables.empty?
62: notifies :run, “execute[clear the gitlab-rails cache]”, :immediately
63: dependent_services.each do |svc|
64: notifies :restart, svc, :immediately
65: end
66: not_if “(test -f #{db_migrate_status_file}) && (cat #{db_migrate_status_file} | grep -Fx 0)”
67: only_if { node[‘gitlab’][‘gitlab-rails’][‘auto_migrate’] }
68: endCompiled Resource:
Declared in /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/database_migrations.rb:51:in `from_file’
bash(“migrate gitlab-rails database”) do
action [:run]
retries 0
retry_delay 2
default_guard_interpreter :default
command “migrate gitlab-rails database”
backup 5
returns 0
code " set -e\n log_file="/var/log/gitlab/gitlab-rails/gitlab-rails-db-migrate-(date +%Y-%m-%d-%H-%M-%S).log\"\n umask 077\n /opt/gitlab/bin/gitlab-rake gitlab:db:configure 2>& 1 | tee {log_file}\n STATUS=${PIPESTATUS[0]}\n echo $STATUS > /var/opt/gitlab/gitlab-rails/upgrade-status/db-migrate-873248b1f0d3a7a5535771a3a1635803-16502c8\n exit $STATUS\n"
interpreter “bash”
declared_type :bash
cookbook_name “gitlab”
recipe_name “database_migrations”
not_if “(test -f /var/opt/gitlab/gitlab-rails/upgrade-status/db-migrate-873248b1f0d3a7a5535771a3a1635803-16502c8) && (cat /var/opt/gitlab/gitlab-rails/upgrade-status/db-migrate-873248b1f0d3a7a5535771a3a1635803-16502c8 | grep -Fx 0)”
only_if { #code block }
endPlatform:
x86_64-linux
Recipe: gitlab::gitlab-rails
-
execute[clear the gitlab-rails cache] action run
[execute] rake aborted!
NoMethodError: undefined methodeach' for nil:NilClass /opt/gitlab/embedded/service/gitlab-rails/config/initializers/1_settings.rb:143:in
<top (required)>’
/opt/gitlab/embedded/service/gitlab-rails/config/environment.rb:5:in<top (required)>' /opt/gitlab/embedded/bin/bundle:23:in
load’
/opt/gitlab/embedded/bin/bundle:23:in `’
Tasks: TOP => cache:clear => cache:clear:redis => environment
(See full trace by running task with --trace)================================================================================
Error executing actionrun
on resource ‘execute[clear the gitlab-rails cache]’Mixlib::ShellOut::ShellCommandFailed
Expected process to exit with [0], but received ‘1’
---- Begin output of /opt/gitlab/bin/gitlab-rake cache:clear ----
STDOUT:
STDERR: rake aborted!
NoMethodError: undefined methodeach' for nil:NilClass /opt/gitlab/embedded/service/gitlab-rails/config/initializers/1_settings.rb:143:in
<top (required)>’
/opt/gitlab/embedded/service/gitlab-rails/config/environment.rb:5:in<top (required)>' /opt/gitlab/embedded/bin/bundle:23:in
load’
/opt/gitlab/embedded/bin/bundle:23:in `’
Tasks: TOP => cache:clear => cache:clear:redis => environment
(See full trace by running task with --trace)
---- End output of /opt/gitlab/bin/gitlab-rake cache:clear ----
Ran /opt/gitlab/bin/gitlab-rake cache:clear returned 1Resource Declaration:
In /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/gitlab-rails.rb
367: execute “clear the gitlab-rails cache” do
368: command “/opt/gitlab/bin/gitlab-rake cache:clear”
369: action :nothing
370: not_if { omnibus_helper.not_listening?(‘redis’) || !node[‘gitlab’][‘gitlab-rails’][‘rake_cache_clear’] }
371: end
372:Compiled Resource:
Declared in /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/gitlab-rails.rb:367:in `from_file’
execute(“clear the gitlab-rails cache”) do
action [:nothing]
retries 0
retry_delay 2
default_guard_interpreter :execute
command “/opt/gitlab/bin/gitlab-rake cache:clear”
backup 5
returns 0
declared_type :execute
cookbook_name “gitlab”
recipe_name “gitlab-rails”
not_if { #code block }
endPlatform:
x86_64-linux
Running handlers:
Running handlers complete
Chef Client failed. 8 resources updated in 49 seconds
Checking for an omnibus managed postgresql: OK
Checking for a newer version of PostgreSQL to install
No new version of PostgreSQL installed, nothing to upgrade to
Ensuring PostgreSQL is updated: OK
Restarting previously running GitLab services
ok: run: gitaly: (pid 12243) 0s
ok: run: gitlab-monitor: (pid 12253) 0s
ok: run: gitlab-workhorse: (pid 12257) 1s
ok: run: logrotate: (pid 12269) 0s
ok: run: mattermost: (pid 12275) 1s
ok: run: nginx: (pid 12283) 0s
ok: run: node-exporter: (pid 12291) 1s
ok: run: postgres-exporter: (pid 12301) 0s
ok: run: postgresql: (pid 11657) 57s
ok: run: prometheus: (pid 12309) 0s
ok: run: redis: (pid 11665) 57s
ok: run: redis-exporter: (pid 12318) 1s
ok: run: sidekiq: (pid 12327) 0s
ok: run: unicorn: (pid 12334) 1s
_______ __ __ __
/ ____(_) /_/ / ____ _/ /_
/ / / / / / / __ `/ __
/ // / / // // // / // /
_//_/___/_,/./
Upgrade complete! If your GitLab server is misbehaving try running
sudo gitlab-ctl restart
before anything else.
If you need to roll back to the previous version you can use the database
backup made during the upgrade (scroll up for the filename).
csgadmin@gitlab:~$ sudo gitlab-ctl restart
ok: run: gitaly: (pid 12762) 0s
ok: run: gitlab-monitor: (pid 12779) 1s
ok: run: gitlab-workhorse: (pid 12783) 0s
ok: run: logrotate: (pid 12792) 1s
ok: run: mattermost: (pid 12806) 0s
ok: run: nginx: (pid 12812) 0s
ok: run: node-exporter: (pid 12824) 1s
ok: run: postgres-exporter: (pid 12831) 0s
ok: run: postgresql: (pid 12840) 1s
ok: run: prometheus: (pid 12848) 0s
ok: run: redis: (pid 12855) 1s
ok: run: redis-exporter: (pid 12860) 0s
ok: run: sidekiq: (pid 12868) 0s
ok: run: unicorn: (pid 12877) 1s