Hello,
I’m trying to run the Gitlab server as a docker container with Blue-Green Deployment.
I want to use AWS ECS, which uses Blue-Green Deployment. If I run a new version, ECS run the old and new version simultaneously for some time. If health checks pass successfully, the old version of the container will be terminated. It works with a lot of applications but not with Gitlab.
I tried it on my local computer using these commands:
mkdir -p /tmp/gitlab/{config,logs,data}
sudo docker run --hostname gitlab.example.com -v /tmp/gitlab/config:/etc/gitlab -v /tmp/gitlab/logs:/var/log/gitlab -v /tmp/gitlab/data:/var/opt/gitlab --shm-size 256m gitlab/gitlab-ce:16.1.2-ce.0
sudo docker run --hostname gitlab.example.com -v /tmp/gitlab/config:/etc/gitlab -v /tmp/gitlab/logs:/var/log/gitlab -v /tmp/gitlab/data:/var/opt/gitlab --shm-size 256m gitlab/gitlab-ce:16.1.2-ce.0
Both containers have the same (shared) storage for Blue-Green Deployment simulation. The first container ran without any problem. But the second container always crashes during startup.
The second container output:
...
* ruby_block[wait for logrotate service socket] action run (skipped due to not_if)
* file[/var/log/gitlab/logrotate/current] action touch (skipped due to only_if)
================================================================================
Error executing action `run` on resource 'ruby_block[restart_log_service]'
================================================================================
Mixlib::ShellOut::ShellCommandFailed
------------------------------------
Expected process to exit with [0], but received '1'
---- Begin output of /opt/gitlab/embedded/bin/sv restart /opt/gitlab/service/logrotate/log ----
STDOUT: timeout: down: /opt/gitlab/service/logrotate/log: 1s, normally up, want up
STDERR:
---- End output of /opt/gitlab/embedded/bin/sv restart /opt/gitlab/service/logrotate/log ----
Ran /opt/gitlab/embedded/bin/sv restart /opt/gitlab/service/logrotate/log returned 1
Cookbook Trace: (most recent call first)
----------------------------------------
/opt/gitlab/embedded/cookbooks/cache/cookbooks/runit/libraries/helpers.rb:136:in `safe_sv_shellout!'
/opt/gitlab/embedded/cookbooks/cache/cookbooks/runit/libraries/helpers.rb:168:in `restart_log_service'
/opt/gitlab/embedded/cookbooks/cache/cookbooks/runit/libraries/provider_runit_service.rb:69:in `block (3 levels) in <class:RunitService>'
/opt/gitlab/embedded/cookbooks/cache/cookbooks/runit/libraries/provider_runit_service.rb:296:in `block in <class:RunitService>'
Resource Declaration:
---------------------
# In /opt/gitlab/embedded/cookbooks/cache/cookbooks/runit/libraries/provider_runit_service.rb
66: ruby_block 'restart_log_service' do
67: block do
68: action_enable
69: restart_log_service
70: end
71: action :nothing
72: end
73:
Compiled Resource:
------------------
# Declared in /opt/gitlab/embedded/cookbooks/cache/cookbooks/runit/libraries/provider_runit_service.rb:66:in `block in <class:RunitService>'
ruby_block("restart_log_service") do
action [:nothing]
default_guard_interpreter :default
declared_type :ruby_block
cookbook_name "logrotate"
recipe_name "enable"
block #<Proc:0x00007f6fbfa4b8d0 /opt/gitlab/embedded/cookbooks/cache/cookbooks/runit/libraries/provider_runit_service.rb:67>
end
System Info:
------------
chef_version=17.10.0
platform=ubuntu
platform_version=22.04
ruby=ruby 3.0.6p216 (2023-03-30 revision 23a532679b) [x86_64-linux]
program_name=/opt/gitlab/embedded/bin/cinc-client
executable=/opt/gitlab/embedded/bin/cinc-client
================================================================================
Error executing action `enable` on resource 'runit_service[logrotate]'
================================================================================
Mixlib::ShellOut::ShellCommandFailed
------------------------------------
ruby_block[restart_log_service] (logrotate::enable line 66) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
---- Begin output of /opt/gitlab/embedded/bin/sv restart /opt/gitlab/service/logrotate/log ----
STDOUT: timeout: down: /opt/gitlab/service/logrotate/log: 1s, normally up, want up
STDERR:
---- End output of /opt/gitlab/embedded/bin/sv restart /opt/gitlab/service/logrotate/log ----
Ran /opt/gitlab/embedded/bin/sv restart /opt/gitlab/service/logrotate/log returned 1
Cookbook Trace: (most recent call first)
----------------------------------------
/opt/gitlab/embedded/cookbooks/cache/cookbooks/runit/libraries/helpers.rb:136:in `safe_sv_shellout!'
/opt/gitlab/embedded/cookbooks/cache/cookbooks/runit/libraries/helpers.rb:168:in `restart_log_service'
/opt/gitlab/embedded/cookbooks/cache/cookbooks/runit/libraries/provider_runit_service.rb:69:in `block (3 levels) in <class:RunitService>'
/opt/gitlab/embedded/cookbooks/cache/cookbooks/runit/libraries/provider_runit_service.rb:296:in `block in <class:RunitService>'
Resource Declaration:
---------------------
# In /opt/gitlab/embedded/cookbooks/cache/cookbooks/logrotate/recipes/enable.rb
21: runit_service "logrotate" do
22: start_down node['logrotate']['ha']
23: control ['t']
24: options({
25: log_directory: logging_settings[:log_directory],
26: log_user: logging_settings[:runit_owner],
27: log_group: logging_settings[:runit_group],
28: }.merge(params))
29: log_options logging_settings[:options]
30: end
31:
Compiled Resource:
------------------
# Declared in /opt/gitlab/embedded/cookbooks/cache/cookbooks/logrotate/recipes/enable.rb:21:in `from_file'
runit_service("logrotate") do
action [:enable]
updated true
updated_by_last_action true
default_guard_interpreter :default
declared_type :runit_service
cookbook_name "logrotate"
recipe_name "enable"
env {}
service_name "logrotate"
options {:env_dir=>"/opt/gitlab/sv/logrotate/env", :log_directory=>"/var/log/gitlab/logrotate", :log_user=>"root", :log_group=>nil}
owner "root"
group "root"
supports {:restart=>nil, :reload=>nil, :status=>nil}
start_command_name "start"
stop_command_name "stop"
restart_command_name "restart"
status_command_name "status"
start_down false
control ["t"]
log_options {"svlogd_size"=>209715200, "svlogd_num"=>30, "svlogd_timeout"=>86400, "svlogd_filter"=>"gzip", "svlogd_udp"=>nil, "svlogd_prefix"=>nil, "udp_log_shipping_host"=>nil, "udp_log_shipping_hostname"=>nil, "udp_log_shipping_port"=>514, "logrotate_frequency"=>"daily", "logrotate_maxsize"=>nil, "logrotate_size"=>nil, "logrotate_rotate"=>30, "logrotate_compress"=>"compress", "logrotate_method"=>"copytruncate", "logrotate_postrotate"=>nil, "logrotate_dateformat"=>nil, "log_group"=>nil, "enable"=>true, "ha"=>false, "dir"=>"/var/opt/gitlab/logrotate", "log_directory"=>"/var/log/gitlab/logrotate", "services"=>["nginx", "puma", "gitlab-rails", "gitlab-shell", "gitlab-workhorse", "gitlab-pages", "gitlab-kas", "gitaly", "mailroom"], "pre_sleep"=>600, "post_sleep"=>3000}
run_template_name "logrotate"
log_template_name "logrotate"
check_script_template_name "logrotate"
finish_script_template_name "logrotate"
control_template_names {"t"=>"logrotate"}
end
System Info:
------------
chef_version=17.10.0
platform=ubuntu
platform_version=22.04
ruby=ruby 3.0.6p216 (2023-03-30 revision 23a532679b) [x86_64-linux]
program_name=/opt/gitlab/embedded/bin/cinc-client
executable=/opt/gitlab/embedded/bin/cinc-client
[2023-07-20T18:33:56+00:00] INFO: Running queued delayed notifications before re-raising exception
[2023-07-20T18:33:56+00:00] INFO: templatesymlink[Create a gitlab.yml and create a symlink to Rails root] sending run action to execute[clear the gitlab-rails cache] (delayed)
Recipe: gitlab::gitlab-rails
* execute[clear the gitlab-rails cache] action run[2023-07-20T18:33:56+00:00] INFO: /opt/gitlab/init/redis status does not exist.
(skipped due to not_if)
Running handlers:
[2023-07-20T18:33:56+00:00] ERROR: Running exception handlers
There was an error running gitlab-ctl reconfigure:
runit_service[logrotate] (logrotate::enable line 21) had an error: Mixlib::ShellOut::ShellCommandFailed: ruby_block[restart_log_service] (logrotate::enable line 66) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
---- Begin output of /opt/gitlab/embedded/bin/sv restart /opt/gitlab/service/logrotate/log ----
STDOUT: timeout: down: /opt/gitlab/service/logrotate/log: 1s, normally up, want up
STDERR:
---- End output of /opt/gitlab/embedded/bin/sv restart /opt/gitlab/service/logrotate/log ----
Ran /opt/gitlab/embedded/bin/sv restart /opt/gitlab/service/logrotate/log returned 1
Running handlers complete
[2023-07-20T18:33:56+00:00] ERROR: Exception handlers complete
Infra Phase failed. 61 resources updated in 40 seconds
[2023-07-20T18:33:56+00:00] FATAL: Stacktrace dumped to /opt/gitlab/embedded/cookbooks/cache/cinc-stacktrace.out
[2023-07-20T18:33:56+00:00] FATAL: ---------------------------------------------------------------------------------------
[2023-07-20T18:33:56+00:00] FATAL: PLEASE PROVIDE THE CONTENTS OF THE stacktrace.out FILE (above) IF YOU FILE A BUG REPORT
[2023-07-20T18:33:56+00:00] FATAL: ---------------------------------------------------------------------------------------
[2023-07-20T18:33:56+00:00] FATAL: Mixlib::ShellOut::ShellCommandFailed: runit_service[logrotate] (logrotate::enable line 21) had an error: Mixlib::ShellOut::ShellCommandFailed: ruby_block[restart_log_service] (logrotate::enable line 66) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
---- Begin output of /opt/gitlab/embedded/bin/sv restart /opt/gitlab/service/logrotate/log ----
STDOUT: timeout: down: /opt/gitlab/service/logrotate/log: 1s, normally up, want up
STDERR:
---- End output of /opt/gitlab/embedded/bin/sv restart /opt/gitlab/service/logrotate/log ----
Ran /opt/gitlab/embedded/bin/sv restart /opt/gitlab/service/logrotate/log returned 1
Can I use Blue-Green Deployment for the Gitlab server? And how?