No such file or directory @ realpath_rec - /opt/gitlab/sv/redis/log/config during upgrade of docker images

cometchaser · June 23, 2020, 7:27pm

Hi all,

I have some troubles to update my gitlab instance (docker) from 12.4 to the newest version.
First I change the version directly to 13.1, which fails. Afterwards I found in the docs, that the recommendation is to update to the latest minor version 12.10 first. So, I tried to run the gitlab/gitlab-ce:12.10.11-ce.0 image. But when starting the reconfigure fails

Running handlers:
There was an error running gitlab-ctl reconfigure:

runit_service[redis] (redis::enable line 66) had an error: Errno::ENOENT: template[/var/log/gitlab/redis   /config] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/runit/libraries/provider_runit_service.rb line 136) had an error: Errno::ENOENT: No such file or directory @ realpath_rec - /opt/gitlab/sv/redis/log/config

Running handlers complete

Chef Client failed. 0 resources updated in 16 seconds

Any ideas?

Recipe: redis::enable

runit_service[redis] action enable
- ruby_block[restart_service] action nothing (skipped due to action :nothing)
- ruby_block[restart_log_service] action nothing (skipped due to action :nothing)
- ruby_block[reload_log_service] action nothing (skipped due to action :nothing)
- directory[/opt/gitlab/sv/redis] action create (up to date)
- template[/opt/gitlab/sv/redis/run] action create (up to date)
- directory[/opt/gitlab/sv/redis/log] action create (up to date)
- directory[/opt/gitlab/sv/redis/log/main] action create (up to date)
- template[/opt/gitlab/sv/redis/log/run] action create (up to date)
- template[/var/log/gitlab/redis/config] action create
  
  ================================================================================
  Error executing action create on resource ‘template[/var/log/gitlab/redis/config]’
  
  Errno::ENOENT
  
  No such file or directory @ realpath_rec - /opt/gitlab/sv/redis/log/config
  
  Cookbook Trace:
  
  /opt/gitlab/embedded/cookbooks/cache/cookbooks/runit/libraries/provider_runit_service.rb:255:in `block in class:RunitService’
  
  Resource Declaration:
  
  In /opt/gitlab/embedded/cookbooks/cache/cookbooks/runit/libraries/provider_runit_service.rb
  
  136: template “#{new_resource.options[:log_directory]}/config” do
  137: owner new_resource.owner unless new_resource.owner.nil?
  138: group new_resource.group unless new_resource.group.nil?
  139: mode ‘0644’
  140: action :create
  141: cookbook template_cookbook
  142: source “sv-#{new_resource.log_template_name}-log-config.erb”
  143: variables new_resource.log_options
  144: notifies :create, ‘ruby_block[reload_log_service]’
  145: notifies :create, “ruby_block[verify_chown_persisted_on_#{new_resource.name}]”, :immediately
  146: end
  147:
  
  Compiled Resource:
  
  Declared in /opt/gitlab/embedded/cookbooks/cache/cookbooks/runit/libraries/provider_runit_service.rb:136:in `block in class:RunitService’
  
  template("/var/log/gitlab/redis/config") do
  action [:create]
  default_guard_interpreter :default
  source “sv-redis-log-config.erb”
  declared_type :template
  cookbook_name “redis”
  mode “0644”
  variables {“svlogd_size”=>209715200, “svlogd_num”=>30, “svlogd_timeout”=>86400, “svlogd_filter”=>“gzip”, “svlogd_udp”=>nil, “svlogd_prefix”=>nil, “udp_log_shipping_host”=>nil, “udp_log_shipping_hostname”=>nil, “udp_log_shipping_port”=>514, “logrotate_frequency”=>“daily”, “logrotate_maxsize”=>nil, “logrotate_size”=>nil, “logrotate_rotate”=>30, “logrotate_compress”=>“compress”, “logrotate_method”=>“copytruncate”, “logrotate_postrotate”=>nil, “logrotate_dateformat”=>nil, “enable”=>true, “ha”=>false, “hz”=>10, “dir”=>"/var/opt/gitlab/redis", “log_directory”=>"/var/log/gitlab/redis", “username”=>“gitlab-redis”, “group”=>“gitlab-redis”, “uid”=>nil, “gid”=>nil, “shell”=>"/bin/false", “home”=>"/var/opt/gitlab/redis", “bind”=>“127.0.0.1”, “port”=>0, “maxclients”=>“10000”, “maxmemory”=>“0”, “maxmemory_policy”=>“noeviction”, “maxmemory_samples”=>5, “tcp_backlog”=>511, “tcp_timeout”=>60, “tcp_keepalive”=>300, “password”=>nil, “unixsocket”=>"/var/opt/gitlab/redis/redis.socket", “unixsocketperm”=>“777”, “master”=>true, “master_name”=>“gitlab-redis”, “master_ip”=>nil, “master_port”=>6379, “master_password”=>nil, “client_output_buffer_limit_normal”=>“0 0 0”, “client_output_buffer_limit_slave”=>“256mb 64mb 60”, “client_output_buffer_limit_pubsub”=>“32mb 8mb 60”, “save”=>[“900 1”, “300 10”, “60 10000”], “rename_commands”=>{“KEYS”=>""}, “announce_ip”=>nil, “announce_port”=>nil}
  owner “root”
  group “root”
  path “/var/log/gitlab/redis/config”
  verifications
  cookbook “redis”
  end
  
  System Info:
  
  chef_version=14.14.29
  platform=ubuntu
  platform_version=16.04
  ruby=ruby 2.6.6p146 (2020-03-31 revision 67876) [x86_64-linux]
  program_name=/opt/gitlab/embedded/bin/chef-client
  executable=/opt/gitlab/embedded/bin/chef-client
================================================================================
Error executing action enable on resource ‘runit_service[redis]’

Errno::ENOENT

template[/var/log/gitlab/redis/config] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/runit/libraries/provider_runit_service.rb line 136) had an error: Errno::ENOENT: No such file or directory @ realpath_rec - /opt/gitlab/sv/redis/log/config

Cookbook Trace:

/opt/gitlab/embedded/cookbooks/cache/cookbooks/runit/libraries/provider_runit_service.rb:255:in `block in class:RunitService’

Resource Declaration:

In /opt/gitlab/embedded/cookbooks/cache/cookbooks/redis/definitions/redis_service.rb

66: runit_service ‘redis’ do
67: down node[‘redis’][‘ha’]
68: template_name ‘redis’
69: options({
70: service: ‘redis’,
71: log_directory: redis_log_dir
72: }.merge(params))
73: log_options node[‘gitlab’][‘logging’].to_hash.merge(node[‘redis’].to_hash)
74: end
75:
76: if node[‘gitlab’][‘bootstrap’][‘enable’]
77: execute “/opt/gitlab/bin/gitlab-ctl start redis” do
78: retries 20
79: end
80: end
81:

Compiled Resource:

Declared in /opt/gitlab/embedded/cookbooks/cache/cookbooks/redis/definitions/redis_service.rb:66:in `block in from_file’

runit_service(“redis”) do
params {:socket_group=>“git”, :name=>nil}
provider Chef::Provider::RunitService
action :enable
default_guard_interpreter :default
service_name “redis”
enabled false
running false
masked nil
options {:service=>“redis”, :log_directory=>"/var/log/gitlab/redis", :socket_group=>“git”, :name=>nil}
pattern “redis”
start_command “start”
stop_command “stop”
status_command “status”
restart_command “restart”
supports {:restart=>true, :reload=>true, :status=>true}
sv_bin “/opt/gitlab/embedded/bin/sv”
sv_dir “/opt/gitlab/sv”
service_dir “/opt/gitlab/service”
lsb_init_dir “/opt/gitlab/init”
log true
owner “root”
group “root”
restart_on_update true
run_template_name “redis”
log_template_name “redis”
check_script_template_name “redis”
finish_script_template_name “redis”
sv_templates true
log_options {“svlogd_size”=>209715200, “svlogd_num”=>30, “svlogd_timeout”=>86400, “svlogd_filter”=>“gzip”, “svlogd_udp”=>nil, “svlogd_prefix”=>nil, “udp_log_shipping_host”=>nil, “udp_log_shipping_hostname”=>nil, “udp_log_shipping_port”=>514, “logrotate_frequency”=>“daily”, “logrotate_maxsize”=>nil, “logrotate_size”=>nil, “logrotate_rotate”=>30, “logrotate_compress”=>“compress”, “logrotate_method”=>“copytruncate”, “logrotate_postrotate”=>nil, “logrotate_dateformat”=>nil, “enable”=>true, “ha”=>false, “hz”=>10, “dir”=>"/var/opt/gitlab/redis", “log_directory”=>"/var/log/gitlab/redis", “username”=>“gitlab-redis”, “group”=>“gitlab-redis”, “uid”=>nil, “gid”=>nil, “shell”=>"/bin/false", “home”=>"/var/opt/gitlab/redis", “bind”=>“127.0.0.1”, “port”=>0, “maxclients”=>“10000”, “maxmemory”=>“0”, “maxmemory_policy”=>“noeviction”, “maxmemory_samples”=>5, “tcp_backlog”=>511, “tcp_timeout”=>60, “tcp_keepalive”=>300, “password”=>nil, “unixsocket”=>"/var/opt/gitlab/redis/redis.socket", “unixsocketperm”=>“777”, “master”=>true, “master_name”=>“gitlab-redis”, “master_ip”=>nil, “master_port”=>6379, “master_password”=>nil, “client_output_buffer_limit_normal”=>“0 0 0”, “client_output_buffer_limit_slave”=>“256mb 64mb 60”, “client_output_buffer_limit_pubsub”=>“32mb 8mb 60”, “save”=>[“900 1”, “300 10”, “60 10000”], “rename_commands”=>{“KEYS”=>""}, “announce_ip”=>nil, “announce_port”=>nil}
declared_type :runit_service
cookbook_name “redis”
recipe_name “enable”
service_mirror # Declared in

service(“redis”) do
provider Chef::Provider::Service::Simple
action [:nothing]
default_guard_interpreter :default
service_name “redis”
enabled nil
running nil
masked nil
pattern “redis”
start_command “/opt/gitlab/embedded/bin/chpst -u root:root /opt/gitlab/embedded/bin/sv start /opt/gitlab/service/redis”
stop_command “/opt/gitlab/embedded/bin/chpst -u root:root /opt/gitlab/embedded/bin/sv stop /opt/gitlab/service/redis”
status_command “/opt/gitlab/embedded/bin/chpst -u root:root /opt/gitlab/embedded/bin/sv status /opt/gitlab/service/redis”
restart_command “/opt/gitlab/embedded/bin/chpst -u root:root /opt/gitlab/embedded/bin/sv restart /opt/gitlab/service/redis”
supports {:restart=>true, :reload=>true, :status=>true}
end

end

System Info:

chef_version=14.14.29
platform=ubuntu
platform_version=16.04
ruby=ruby 2.6.6p146 (2020-03-31 revision 67876) [x86_64-linux]
program_name=/opt/gitlab/embedded/bin/chef-client
executable=/opt/gitlab/embedded/bin/chef-client

robpou · June 24, 2020, 3:23pm

Hi there. I have the exact same issue after the exact same “error”. (I knew that I can’t directly go to 13.x but thought I already updated this instance to 13.0 too like the other one and I’m about to do a minor update to 13.1).

The problem seems to be painful in the sense that even if I create the Redis configfile by hand, the gitlab-ctl reconfigure will simply die at the next step, which is gitaly.

Any help would be appreciated.

robpou · June 24, 2020, 3:30pm

Erm… okay. This is a bit ugly. I had to create the files /opt/gitlab/sv/gitaly/log/config and /opt/gitlab/sv/postgresql/log/config after the redis one and now the reconfigure went through. Gitlab is up and running with 12.7.x. Now I’m about to do the 12.10 upgrade and try the 13.0 upgrade.

Let’s see if I have to play with the configs every time or they will work after the first workaround (which I doubt).

The content is every time the same:

s209715200
n30
t86400
!gzip

robpou · June 24, 2020, 4:54pm

Interesting. So, like I was expecting it, the files had to be re-created after the 12.10 pull. But afterwards, and by that I mean latest (13.1.0 as of writing this) managed to start without any external help.

That’s ugly and all but seems to be fixed in the 13.x branch. Not in the 12.x though.

Marshalleq · June 24, 2020, 10:19pm

I’ve gotten this overnight too, which is weird because I keep my dockers up to date and I don’t think there’s much possibility I would have skipped a version anywhere. The end result is I can’t log in to anything.

Anyway, to fix it I rolled my docker image back to an appropriate one to ensure the DB was prior to the upgrade using ZFS then pulled the specific version gitlab/gitlab-ce:12.10.11-ce.0.

I’ve found that from 12.10 I can go to 13.06 but cannot go further, I assume without some manual intervention.

Ideally, I should get to 13.1.1

jacobnollette · June 25, 2020, 12:40am

We’re having this issue. We can’t get the docker container to run the migration after reboot. Any ideas?

conrad784 · June 25, 2020, 11:40am

Like robpou said the files in /opt/gitlab/sv/{redis,gitaly,postgresql}/log/config are missing.

I started version gitlab-ee:12.10.1-ee.0 and executed

docker exec <containername> cp /var/opt/gitlab/redis/redis.conf /opt/gitlab/sv/redis/log/config
docker exec <containername> cp /var/opt/gitlab/gitaly/config.toml /opt/gitlab/sv/gitaly/log/config
docker exec <containername> cp /var/opt/gitlab/postgresql/data/postgresql.conf /opt/gitlab/sv/postgresql/log/config

Then gitlab is starting again.
Afterwards I was able to upgrade to gitlab-ee:13.1.1-ee.0

cometchaser · June 25, 2020, 7:22pm

Thanks this works!

mburger · July 6, 2020, 6:19pm

Thx it worked also for me

joamag · July 8, 2020, 9:54am

It also worked for me. @conrad784 thank you!

cobba · July 9, 2020, 8:57pm

Hi,
i’m facing the same problem. I tried to update from 12.2.1-ce to 13.1.4-ce.0 and it failed.
Now i can’t get it up and running again. Not even with the old image.
I’m using the gitlab-ce docker images and the container stops immediatly after the error.
Any suggestions how i can get the copy commands from the solutions in there?

cobba · July 9, 2020, 9:21pm

I found an article [https://7thzero.com/blog/ad-hoc-repairs-to-a-failed-gitlab-ce-upgrade-12-8-greater-than-13-0-8](http://Ad-hoc repairs to a failed gitlab-ce upgrade (12.8 -> 13.0.8))
That seems to be helping with my problem.

marcel.pokrandt · July 23, 2020, 12:40pm

The article helped me…

My container didn´t start anymore.
I “simply” put empty “config” files in the folders like in the article (“simply” took me ~3 hours).

# touch config
# docker cp config f0a53fbfc5ff07:/opt/gitlab/sv/redis/log
# docker cp config f0a53fbfc5ff07:/var/log/gitlab/gitaly
# docker cp config f0a53fbfc5ff07:/opt/gitlab/sv/gitaly/log
# docker cp config f0a53fbfc5ff07:/opt/gitlab/sv/postgresql/log

Then the migrations ran successfully (version 12.10.1-ce.0) and migrated Postgres tables and so on.

But… I still have a problem: when I destroy the container and restart it, the problem with the missing config files reoccurs

marcel.pokrandt · July 23, 2020, 1:02pm

Follow up on my last statement:

I migrated then from 12.10.1 to ‘gitlab/gitlab-ce:13.0.10-ce.0’ where I had to manually add the empty files again and THEN migrated to gitlab/gitlab-ce:13.2.0-ce.0 … wich worked without adding the files.

The container now works after destroying and recreation without manual input.

LEARNING:
Don´t use “:latest” tag in docker descriptors. Always use a specific version. Do backups before upgrading

olstjos · August 14, 2020, 4:56pm

conrad784, thank you so much, my gitlab docker container is back up and running!

gotjoshua · August 25, 2020, 7:17pm

I needed to alter this slightly (as the directories were also missing):

docker-compose exec gitlab bash -c 'mkdir -p /opt/gitlab/sv/redis/log/ && cp /var/opt/gitlab/redis/redis.conf /opt/gitlab/sv/redis/log/config'
docker-compose exec gitlab bash -c 'mkdir -p /opt/gitlab/sv/gitaly/log/ && cp /var/opt/gitlab/gitaly/config.toml /opt/gitlab/sv/gitaly/log/config'
docker-compose exec gitlab bash -c 'mkdir -p /opt/gitlab/sv/postgresql/log/ && cp /var/opt/gitlab/postgresql/data/postgresql.conf /opt/gitlab/sv/postgresql/log/config'

azriel49 · November 17, 2020, 2:34pm

You save my day ! Thank you

Topic		Replies	Views
Inherited Gitlab-ee on docker swarm. Errors after trying to upgrade Self-managed docker	0	503	August 13, 2021
Docker compose update error: Error executing action `enable` on resource 'runit_service[redis]' Upgrade	0	1198	July 15, 2020
Failed to upgrade GitLab ee 14.9.4 --> 14.10.X (Docker) Upgrade docker	3	1994	June 13, 2022
Gitlab update - error "permission denied" Upgrade	5	80	December 6, 2024
Docker Container Upgrade Issue Upgrade docker	1	746	June 8, 2023

No such file or directory @ realpath_rec - /opt/gitlab/sv/redis/log/config during upgrade of docker images

================================================================================
Error executing action `create` on resource ‘template[/var/log/gitlab/redis/config]’

Errno::ENOENT

Cookbook Trace:

Resource Declaration:

In /opt/gitlab/embedded/cookbooks/cache/cookbooks/runit/libraries/provider_runit_service.rb

Compiled Resource:

Declared in /opt/gitlab/embedded/cookbooks/cache/cookbooks/runit/libraries/provider_runit_service.rb:136:in `block in class:RunitService’

System Info:

================================================================================
Error executing action `enable` on resource ‘runit_service[redis]’

Errno::ENOENT

Cookbook Trace:

Resource Declaration:

In /opt/gitlab/embedded/cookbooks/cache/cookbooks/redis/definitions/redis_service.rb

Compiled Resource:

Declared in /opt/gitlab/embedded/cookbooks/cache/cookbooks/redis/definitions/redis_service.rb:66:in `block in from_file’

System Info:

No such file or directory @ realpath_rec - /opt/gitlab/sv/redis/log/config during upgrade of docker images

================================================================================ Error executing action create on resource ‘template[/var/log/gitlab/redis/config]’

Errno::ENOENT

Cookbook Trace:

Resource Declaration:

In /opt/gitlab/embedded/cookbooks/cache/cookbooks/runit/libraries/provider_runit_service.rb

Compiled Resource:

Declared in /opt/gitlab/embedded/cookbooks/cache/cookbooks/runit/libraries/provider_runit_service.rb:136:in `block in class:RunitService’

System Info:

================================================================================ Error executing action enable on resource ‘runit_service[redis]’

Errno::ENOENT

Cookbook Trace:

Resource Declaration:

In /opt/gitlab/embedded/cookbooks/cache/cookbooks/redis/definitions/redis_service.rb

Compiled Resource:

Declared in /opt/gitlab/embedded/cookbooks/cache/cookbooks/redis/definitions/redis_service.rb:66:in `block in from_file’

System Info:

Related topics

================================================================================
Error executing action `create` on resource ‘template[/var/log/gitlab/redis/config]’

================================================================================
Error executing action `enable` on resource ‘runit_service[redis]’