Hi!
After almost 3 years of upgrading our GitLab CE Omnibus installation successfully (still a very good job, GitLab team! 
) I encountered the first fail today when trying to upgrade from v14.2.5 to v14.3.3.
I’m on a Ubuntu 18.04.6 LTS 64-bit server (kernel: 4.15.0-159-generic) with all available updates applied with apt full-upgrade.
I checked the validity of the upgrade path: https://docs.gitlab.com/ee/update/index.html#upgrade-paths
On the same page in another section https://docs.gitlab.com/ee/update/index.html#1430, it is stated that Ruby 2.7.4 is required. This seems to be ok:
/opt/gitlab/bin/gitlab-ruby -v
ruby 2.7.4p191 (2021-07-07 revision a21a3b7d23) [x86_64-linux]
The command I run to update was: apt install gitlab-ce=14.3.3-ce.0 i.e. I don’t want already upgrade to v14.4.0
This is the error I see:
Recipe: gitlab::gitlab-rails
* execute[clear the gitlab-rails cache] action run
- execute /opt/gitlab/bin/gitlab-rake cache:clear
Recipe: gitaly::enable
* runit_service[gitaly] action restart
================================================================================
Error executing action `restart` on resource 'runit_service[gitaly]'
================================================================================
Mixlib::ShellOut::ShellCommandFailed
------------------------------------
Expected process to exit with [0], but received '1'
---- Begin output of /opt/gitlab/embedded/bin/sv restart /opt/gitlab/service/gitaly ----
STDOUT: timeout: run: /opt/gitlab/service/gitaly: (pid 2036) 1856855s, got TERM
STDERR:
---- End output of /opt/gitlab/embedded/bin/sv restart /opt/gitlab/service/gitaly ----
Ran /opt/gitlab/embedded/bin/sv restart /opt/gitlab/service/gitaly returned 1
Cookbook Trace:
---------------
/opt/gitlab/embedded/cookbooks/cache/cookbooks/runit/libraries/helpers.rb:136:in `tap'
/opt/gitlab/embedded/cookbooks/cache/cookbooks/runit/libraries/helpers.rb:136:in `safe_sv_shellout!'
/opt/gitlab/embedded/cookbooks/cache/cookbooks/runit/libraries/helpers.rb:164:in `restart_service'
/opt/gitlab/embedded/cookbooks/cache/cookbooks/runit/libraries/provider_runit_service.rb:357:in `block in <class:RunitService>'
Resource Declaration:
---------------------
# In /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitaly/recipes/enable.rb
103: runit_service 'gitaly' do
104: start_down node['gitaly']['ha']
105: options({
106: user: account_helper.gitlab_user,
107: groupname: account_helper.gitlab_group,
108: working_dir: working_dir,
109: env_dir: env_directory,
110: bin_path: gitaly_path,
111: wrapper_path: wrapper_path,
112: config_path: config_path,
113: log_directory: log_directory,
114: json_logging: json_logging,
115: open_files_ulimit: open_files_ulimit
116: }.merge(params))
117: log_options node['gitlab']['logging'].to_hash.merge(node['gitaly'].to_hash)
118: end
119:
Compiled Resource:
------------------
# Declared in /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitaly/recipes/enable.rb:103:in `from_file'
runit_service("gitaly") do
action [:enable]
default_guard_interpreter :default
declared_type :runit_service
cookbook_name "gitaly"
recipe_name "enable"
start_down false
service_name "gitaly"
options {:env_dir=>"/opt/gitlab/etc/gitaly/env", :user=>"git", :groupname=>"git", :working_dir=>"/var/opt/gitlab/gitaly", :bin_path=>"/opt/gitlab/embedded/bin/gitaly", :wrapper_path=>"/opt/gitlab/embedded/bin/gitaly-wrapper" , :config_path=>"/var/opt/gitlab/gitaly/config.toml", :log_directory=>"/var/log/gitlab/gitaly", :json_logging=>true, :open_files_ulimit=>15000}
log_options {"svlogd_size"=>209715200, "svlogd_num"=>30, "svlogd_timeout"=>86400, "svlogd_filter"=>"gzip", "svlogd_udp"=>nil, "svlogd_prefix"=>nil, "udp_log_shipping_host"=>nil, "udp_log_shipping_hostname"=>nil, "udp_log_shi pping_port"=>514, "logrotate_frequency"=>"daily", "logrotate_maxsize"=>nil, "logrotate_size"=>nil, "logrotate_rotate"=>30, "logrotate_compress"=>"compress", "logrotate_method"=>"copytruncate", "logrotate_postrotate"=>nil, "logrota te_dateformat"=>nil, "enable"=>true, "ha"=>false, "dir"=>"/var/opt/gitlab/gitaly", "internal_socket_dir"=>"/var/opt/gitlab/gitaly/internal_sockets", "log_directory"=>"/var/log/gitlab/gitaly", "env_directory"=>"/opt/gitlab/etc/gita ly/env", "graceful_restart_timeout"=>nil, "bin_path"=>"/opt/gitlab/embedded/bin/gitaly", "socket_path"=>"/var/opt/gitlab/gitaly/gitaly.socket", "listen_addr"=>nil, "tls_listen_addr"=>nil, "certificate_path"=>nil, "key_path"=>nil, "prometheus_listen_addr"=>"localhost:9236", "logging_level"=>nil, "logging_format"=>"json", "logging_sentry_dsn"=>nil, "logging_ruby_sentry_dsn"=>nil, "logging_sentry_environment"=>nil, "prometheus_grpc_latency_buckets"=>nil, "sto rage"=>[{"name"=>"default", "path"=>"/raid/gitlab/git-data/repositories"}], "auth_token"=>nil, "auth_transitioning"=>false, "git_catfile_cache_size"=>nil, "git_bin_path"=>nil, "open_files_ulimit"=>15000, "ruby_max_rss"=>nil, "ruby _graceful_restart_timeout"=>nil, "ruby_restart_delay"=>nil, "ruby_num_workers"=>nil, "ruby_rugged_git_config_search_path"=>"/opt/gitlab/embedded/etc", "concurrency"=>nil, "daily_maintenance_start_hour"=>nil, "daily_maintenance_sta rt_minute"=>nil, "daily_maintenance_duration"=>nil, "daily_maintenance_storages"=>nil, "daily_maintenance_disabled"=>false, "cgroups_count"=>nil, "cgroups_mountpoint"=>nil, "cgroups_hierarchy_root"=>nil, "cgroups_memory_enabled"=> nil, "cgroups_memory_limit"=>nil, "cgroups_cpu_enabled"=>nil, "cgroups_cpu_shares"=>nil, "pack_objects_cache_enabled"=>false, "pack_objects_cache_dir"=>nil, "pack_objects_cache_max_age"=>nil, "consul_service_name"=>"gitaly", "cust om_hooks_dir"=>nil, "consul_service_meta"=>nil, "env"=>{"HOME"=>"/var/opt/gitlab", "PATH"=>"/opt/gitlab/bin:/opt/gitlab/embedded/bin:/bin:/usr/bin", "TZ"=>":/etc/localtime", "PYTHONPATH"=>"/opt/gitlab/embedded/lib/python3.9/site-p ackages", "ICU_DATA"=>"/opt/gitlab/embedded/share/icu/current", "SSL_CERT_DIR"=>"/opt/gitlab/embedded/ssl/certs/", "GITALY_PID_FILE"=>"/var/opt/gitlab/gitaly/gitaly.pid", "WRAPPER_JSON_LOGGING"=>"true"}}
run_template_name "gitaly"
log_template_name "gitaly"
check_script_template_name "gitaly"
finish_script_template_name "gitaly"
control_template_names {}
end
System Info:
------------
chef_version=15.17.4
platform=ubuntu
platform_version=18.04
ruby=ruby 2.7.4p191 (2021-07-07 revision a21a3b7d23) [x86_64-linux]
program_name=/opt/gitlab/embedded/bin/chef-client
executable=/opt/gitlab/embedded/bin/chef-client
* runit_service[gitaly] action hup
- send hup to runit_service[gitaly]
Recipe: gitlab::gitlab-workhorse
* runit_service[gitlab-workhorse] action restart (up to date)
Recipe: gitlab-pages::enable
* runit_service[gitlab-pages] action restart (up to date)
Recipe: monitoring::gitlab-exporter
* runit_service[gitlab-exporter] action restart (up to date)
Running handlers:
There was an error running gitlab-ctl reconfigure:
runit_service[gitaly] (gitaly::enable line 103) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
---- Begin output of /opt/gitlab/embedded/bin/sv restart /opt/gitlab/service/gitaly ----
STDOUT: timeout: run: /opt/gitlab/service/gitaly: (pid 2036) 1856855s, got TERM
STDERR:
---- End output of /opt/gitlab/embedded/bin/sv restart /opt/gitlab/service/gitaly ----
Ran /opt/gitlab/embedded/bin/sv restart /opt/gitlab/service/gitaly returned 1
Running handlers complete
Chef Infra Client failed. 37 resources updated in 01 minutes 19 seconds
===
There was an error running gitlab-ctl reconfigure. Please check the output above for more
details.
===
dpkg: error processing package gitlab-ce (--configure):
installed gitlab-ce package post-installation script subprocess returned error exit status 1
E: Sub-process /usr/bin/dpkg returned an error code (1)
Can anyone help me to find out what the problem is and how to correctly fix it? Many thanks in advance!
Some more output that might be helpful:
gitlab-ctl status
down: alertmanager: 24425s, normally up; run: log: (pid 2014) 1881191s
run: gitaly: (pid 4970) 24333s; run: log: (pid 2031) 1881191s
run: gitlab-exporter: (pid 4934) 24335s; run: log: (pid 2007) 1881191s
run: gitlab-pages: (pid 4914) 24336s; run: log: (pid 2005) 1881191s
run: gitlab-workhorse: (pid 4892) 24336s; run: log: (pid 2012) 1881191s
down: grafana: 24424s, normally up; run: log: (pid 2016) 1881191s
down: logrotate: 24423s, normally up; run: log: (pid 2013) 1881191s
down: nginx: 24423s, normally up; run: log: (pid 2017) 1881191s
down: node-exporter: 24422s, normally up; run: log: (pid 2015) 1881191s
down: postgres-exporter: 24422s, normally up; run: log: (pid 2035) 1881191s
run: postgresql: (pid 2033) 1881191s; run: log: (pid 2029) 1881191s
down: prometheus: 24422s, normally up; run: log: (pid 2010) 1881191s
down: puma: 24420s, normally up; run: log: (pid 2002) 1881191s
run: redis: (pid 2054) 1881191s; run: log: (pid 2050) 1881191s
down: redis-exporter: 24420s, normally up; run: log: (pid 2009) 1881191s
down: sidekiq: 24416s, normally up; run: log: (pid 2020) 1881191s
gitlab-rake gitlab:check SANITIZE=true
Checking GitLab subtasks ...
Checking GitLab Shell ...
GitLab Shell: ... GitLab Shell version >= 13.21.1 ? ... OK (13.21.1)
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Internal API available: FAILED - Internal API error (502)
gitlab-shell self-check failed
Try fixing it:
Make sure GitLab is running;
Check the gitlab-shell configuration file:
sudo -u git -H editor /opt/gitlab/embedded/service/gitlab-shell/config.yml
Please fix the error above and rerun the checks.
Checking GitLab Shell ... Finished
Checking Gitaly ...
Gitaly: ... default ... OK
Checking Gitaly ... Finished
Checking Sidekiq ...
Sidekiq: ... Running? ... no
Try fixing it:
sudo -u git -H RAILS_ENV=production bin/background_jobs start
For more information see:
doc/install/installation.md in section "Install Init Script"
see log/sidekiq.log for possible errors
Please fix the error above and rerun the checks.
Checking Sidekiq ... Finished
Checking Incoming Email ...
Incoming Email: ... Reply by email is disabled in config/gitlab.yml
Checking Incoming Email ... Finished
Checking LDAP ...
LDAP: ... LDAP is disabled in config/gitlab.yml
Checking LDAP ... Finished
Checking GitLab App ...
Git configured correctly? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... yes
Init script exists? ... skipped (omnibus-gitlab has no init script)
Init script up-to-date? ... skipped (omnibus-gitlab has no init script)
Projects have namespace: ...
2/2 ... yes
2/5 ... yes
2/6 ... yes
2/14 ... yes
2/16 ... yes
2/17 ... yes
2/18 ... yes
2/19 ... yes
2/20 ... yes
2/21 ... yes
2/22 ... yes
2/23 ... yes
2/24 ... yes
2/25 ... yes
2/26 ... yes
2/27 ... yes
2/28 ... yes
2/29 ... yes
2/30 ... yes
2/31 ... yes
2/32 ... yes
2/33 ... yes
2/34 ... yes
2/35 ... yes
2/36 ... yes
2/37 ... yes
2/38 ... yes
2/39 ... yes
2/40 ... yes
2/41 ... yes
2/42 ... yes
2/43 ... yes
2/44 ... yes
2/45 ... yes
2/46 ... yes
2/47 ... yes
2/48 ... yes
2/49 ... yes
2/50 ... yes
2/51 ... yes
2/52 ... yes
2/53 ... yes
2/54 ... yes
2/55 ... yes
2/56 ... yes
2/57 ... yes
2/58 ... yes
2/59 ... yes
2/60 ... yes
2/61 ... yes
2/63 ... yes
2/64 ... yes
2/65 ... yes
Redis version >= 5.0.0? ... yes
Ruby version >= 2.7.2 ? ... yes (2.7.4)
Git version >= 2.31.0 ? ... yes (2.33.0)
Git user has default SSH configuration? ... yes
Active users: ... 12
Is authorized keys file accessible? ... yes
GitLab configured to store new projects in hashed storage? ... yes
All projects are in hashed storage? ... yes
Checking GitLab App ... Finished
Checking GitLab subtasks ... Finished
Best Regards,
-Thomas