No such file or directory @ realpath_rec - /opt/gitlab/sv/redis/log/config during upgrade of docker images

How to Use GitLab Upgrade
,
1

Hi all,

I have some troubles to update my gitlab instance (docker) from 12.4 to the newest version.
First I change the version directly to 13.1, which fails. Afterwards I found in the docs, that the recommendation is to update to the latest minor version 12.10 first. So, I tried to run the gitlab/gitlab-ce:12.10.11-ce.0 image. But when starting the reconfigure fails

Running handlers:
There was an error running gitlab-ctl reconfigure:

runit_service[redis] (redis::enable line 66) had an error: Errno::ENOENT: template[/var/log/gitlab/redis   /config] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/runit/libraries/provider_runit_service.rb line 136) had an error: Errno::ENOENT: No such file or directory @ realpath_rec - /opt/gitlab/sv/redis/log/config

Running handlers complete

Chef Client failed. 0 resources updated in 16 seconds

Any ideas?

Recipe: redis::enable

  • runit_service[redis] action enable

    • ruby_block[restart_service] action nothing (skipped due to action :nothing)

    • ruby_block[restart_log_service] action nothing (skipped due to action :nothing)

    • ruby_block[reload_log_service] action nothing (skipped due to action :nothing)

    • directory[/opt/gitlab/sv/redis] action create (up to date)

    • template[/opt/gitlab/sv/redis/run] action create (up to date)

    • directory[/opt/gitlab/sv/redis/log] action create (up to date)

    • directory[/opt/gitlab/sv/redis/log/main] action create (up to date)

    • template[/opt/gitlab/sv/redis/log/run] action create (up to date)

    • template[/var/log/gitlab/redis/config] action create

      ================================================================================
      Error executing action create on resource ‘template[/var/log/gitlab/redis/config]’

      Errno::ENOENT

      No such file or directory @ realpath_rec - /opt/gitlab/sv/redis/log/config

      Cookbook Trace:

      /opt/gitlab/embedded/cookbooks/cache/cookbooks/runit/libraries/provider_runit_service.rb:255:in `block in class:RunitService’

      Resource Declaration:

      In /opt/gitlab/embedded/cookbooks/cache/cookbooks/runit/libraries/provider_runit_service.rb

      136: template “#{new_resource.options[:log_directory]}/config” do
      137: owner new_resource.owner unless new_resource.owner.nil?
      138: group new_resource.group unless new_resource.group.nil?
      139: mode ‘0644’
      140: action :create
      141: cookbook template_cookbook
      142: source “sv-#{new_resource.log_template_name}-log-config.erb”
      143: variables new_resource.log_options
      144: notifies :create, ‘ruby_block[reload_log_service]’
      145: notifies :create, “ruby_block[verify_chown_persisted_on_#{new_resource.name}]”, :immediately
      146: end
      147:

      Compiled Resource:

      Declared in /opt/gitlab/embedded/cookbooks/cache/cookbooks/runit/libraries/provider_runit_service.rb:136:in `block in class:RunitService’

      template("/var/log/gitlab/redis/config") do
      action [:create]
      default_guard_interpreter :default
      source “sv-redis-log-config.erb”
      declared_type :template
      cookbook_name “redis”
      mode “0644”
      variables {“svlogd_size”=>209715200, “svlogd_num”=>30, “svlogd_timeout”=>86400, “svlogd_filter”=>“gzip”, “svlogd_udp”=>nil, “svlogd_prefix”=>nil, “udp_log_shipping_host”=>nil, “udp_log_shipping_hostname”=>nil, “udp_log_shipping_port”=>514, “logrotate_frequency”=>“daily”, “logrotate_maxsize”=>nil, “logrotate_size”=>nil, “logrotate_rotate”=>30, “logrotate_compress”=>“compress”, “logrotate_method”=>“copytruncate”, “logrotate_postrotate”=>nil, “logrotate_dateformat”=>nil, “enable”=>true, “ha”=>false, “hz”=>10, “dir”=>"/var/opt/gitlab/redis", “log_directory”=>"/var/log/gitlab/redis", “username”=>“gitlab-redis”, “group”=>“gitlab-redis”, “uid”=>nil, “gid”=>nil, “shell”=>"/bin/false", “home”=>"/var/opt/gitlab/redis", “bind”=>“127.0.0.1”, “port”=>0, “maxclients”=>“10000”, “maxmemory”=>“0”, “maxmemory_policy”=>“noeviction”, “maxmemory_samples”=>5, “tcp_backlog”=>511, “tcp_timeout”=>60, “tcp_keepalive”=>300, “password”=>nil, “unixsocket”=>"/var/opt/gitlab/redis/redis.socket", “unixsocketperm”=>“777”, “master”=>true, “master_name”=>“gitlab-redis”, “master_ip”=>nil, “master_port”=>6379, “master_password”=>nil, “client_output_buffer_limit_normal”=>“0 0 0”, “client_output_buffer_limit_slave”=>“256mb 64mb 60”, “client_output_buffer_limit_pubsub”=>“32mb 8mb 60”, “save”=>[“900 1”, “300 10”, “60 10000”], “rename_commands”=>{“KEYS”=>""}, “announce_ip”=>nil, “announce_port”=>nil}
      owner “root”
      group “root”
      path “/var/log/gitlab/redis/config”
      verifications
      cookbook “redis”
      end

      System Info:

      chef_version=14.14.29
      platform=ubuntu
      platform_version=16.04
      ruby=ruby 2.6.6p146 (2020-03-31 revision 67876) [x86_64-linux]
      program_name=/opt/gitlab/embedded/bin/chef-client
      executable=/opt/gitlab/embedded/bin/chef-client

    ================================================================================
    Error executing action enable on resource ‘runit_service[redis]’

    Errno::ENOENT

    template[/var/log/gitlab/redis/config] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/runit/libraries/provider_runit_service.rb line 136) had an error: Errno::ENOENT: No such file or directory @ realpath_rec - /opt/gitlab/sv/redis/log/config

    Cookbook Trace:

    /opt/gitlab/embedded/cookbooks/cache/cookbooks/runit/libraries/provider_runit_service.rb:255:in `block in class:RunitService’

    Resource Declaration:

    In /opt/gitlab/embedded/cookbooks/cache/cookbooks/redis/definitions/redis_service.rb

    66: runit_service ‘redis’ do
    67: down node[‘redis’][‘ha’]
    68: template_name ‘redis’
    69: options({
    70: service: ‘redis’,
    71: log_directory: redis_log_dir
    72: }.merge(params))
    73: log_options node[‘gitlab’][‘logging’].to_hash.merge(node[‘redis’].to_hash)
    74: end
    75:
    76: if node[‘gitlab’][‘bootstrap’][‘enable’]
    77: execute “/opt/gitlab/bin/gitlab-ctl start redis” do
    78: retries 20
    79: end
    80: end
    81:

    Compiled Resource:

    Declared in /opt/gitlab/embedded/cookbooks/cache/cookbooks/redis/definitions/redis_service.rb:66:in `block in from_file’

    runit_service(“redis”) do
    params {:socket_group=>“git”, :name=>nil}
    provider Chef::Provider::RunitService
    action :enable
    default_guard_interpreter :default
    service_name “redis”
    enabled false
    running false
    masked nil
    options {:service=>“redis”, :log_directory=>"/var/log/gitlab/redis", :socket_group=>“git”, :name=>nil}
    pattern “redis”
    start_command “start”
    stop_command “stop”
    status_command “status”
    restart_command “restart”
    supports {:restart=>true, :reload=>true, :status=>true}
    sv_bin “/opt/gitlab/embedded/bin/sv”
    sv_dir “/opt/gitlab/sv”
    service_dir “/opt/gitlab/service”
    lsb_init_dir “/opt/gitlab/init”
    log true
    owner “root”
    group “root”
    restart_on_update true
    run_template_name “redis”
    log_template_name “redis”
    check_script_template_name “redis”
    finish_script_template_name “redis”
    sv_templates true
    log_options {“svlogd_size”=>209715200, “svlogd_num”=>30, “svlogd_timeout”=>86400, “svlogd_filter”=>“gzip”, “svlogd_udp”=>nil, “svlogd_prefix”=>nil, “udp_log_shipping_host”=>nil, “udp_log_shipping_hostname”=>nil, “udp_log_shipping_port”=>514, “logrotate_frequency”=>“daily”, “logrotate_maxsize”=>nil, “logrotate_size”=>nil, “logrotate_rotate”=>30, “logrotate_compress”=>“compress”, “logrotate_method”=>“copytruncate”, “logrotate_postrotate”=>nil, “logrotate_dateformat”=>nil, “enable”=>true, “ha”=>false, “hz”=>10, “dir”=>"/var/opt/gitlab/redis", “log_directory”=>"/var/log/gitlab/redis", “username”=>“gitlab-redis”, “group”=>“gitlab-redis”, “uid”=>nil, “gid”=>nil, “shell”=>"/bin/false", “home”=>"/var/opt/gitlab/redis", “bind”=>“127.0.0.1”, “port”=>0, “maxclients”=>“10000”, “maxmemory”=>“0”, “maxmemory_policy”=>“noeviction”, “maxmemory_samples”=>5, “tcp_backlog”=>511, “tcp_timeout”=>60, “tcp_keepalive”=>300, “password”=>nil, “unixsocket”=>"/var/opt/gitlab/redis/redis.socket", “unixsocketperm”=>“777”, “master”=>true, “master_name”=>“gitlab-redis”, “master_ip”=>nil, “master_port”=>6379, “master_password”=>nil, “client_output_buffer_limit_normal”=>“0 0 0”, “client_output_buffer_limit_slave”=>“256mb 64mb 60”, “client_output_buffer_limit_pubsub”=>“32mb 8mb 60”, “save”=>[“900 1”, “300 10”, “60 10000”], “rename_commands”=>{“KEYS”=>""}, “announce_ip”=>nil, “announce_port”=>nil}
    declared_type :runit_service
    cookbook_name “redis”
    recipe_name “enable”
    service_mirror # Declared in

    service(“redis”) do
    provider Chef::Provider::Service::Simple
    action [:nothing]
    default_guard_interpreter :default
    service_name “redis”
    enabled nil
    running nil
    masked nil
    pattern “redis”
    start_command “/opt/gitlab/embedded/bin/chpst -u root:root /opt/gitlab/embedded/bin/sv start /opt/gitlab/service/redis”
    stop_command “/opt/gitlab/embedded/bin/chpst -u root:root /opt/gitlab/embedded/bin/sv stop /opt/gitlab/service/redis”
    status_command “/opt/gitlab/embedded/bin/chpst -u root:root /opt/gitlab/embedded/bin/sv status /opt/gitlab/service/redis”
    restart_command “/opt/gitlab/embedded/bin/chpst -u root:root /opt/gitlab/embedded/bin/sv restart /opt/gitlab/service/redis”
    supports {:restart=>true, :reload=>true, :status=>true}
    end

    end

    System Info:

    chef_version=14.14.29
    platform=ubuntu
    platform_version=16.04
    ruby=ruby 2.6.6p146 (2020-03-31 revision 67876) [x86_64-linux]
    program_name=/opt/gitlab/embedded/bin/chef-client
    executable=/opt/gitlab/embedded/bin/chef-client

2

Hi there. I have the exact same issue after the exact same “error”. (I knew that I can’t directly go to 13.x but thought I already updated this instance to 13.0 too like the other one and I’m about to do a minor update to 13.1).

The problem seems to be painful in the sense that even if I create the Redis configfile by hand, the gitlab-ctl reconfigure will simply die at the next step, which is gitaly.

Any help would be appreciated.

3

Erm… okay. This is a bit ugly. I had to create the files /opt/gitlab/sv/gitaly/log/config and /opt/gitlab/sv/postgresql/log/config after the redis one and now the reconfigure went through. Gitlab is up and running with 12.7.x. Now I’m about to do the 12.10 upgrade and try the 13.0 upgrade.

Let’s see if I have to play with the configs every time or they will work after the first workaround (which I doubt).

The content is every time the same:

s209715200
n30
t86400
!gzip
4

Interesting. So, like I was expecting it, the files had to be re-created after the 12.10 pull. But afterwards, and by that I mean latest (13.1.0 as of writing this) managed to start without any external help.

That’s ugly and all but seems to be fixed in the 13.x branch. Not in the 12.x though.

5

I’ve gotten this overnight too, which is weird because I keep my dockers up to date and I don’t think there’s much possibility I would have skipped a version anywhere. The end result is I can’t log in to anything.

Anyway, to fix it I rolled my docker image back to an appropriate one to ensure the DB was prior to the upgrade using ZFS then pulled the specific version gitlab/gitlab-ce:12.10.11-ce.0.

I’ve found that from 12.10 I can go to 13.06 but cannot go further, I assume without some manual intervention.

Ideally, I should get to 13.1.1

6

We’re having this issue. We can’t get the docker container to run the migration after reboot. Any ideas?

7

Like robpou said the files in /opt/gitlab/sv/{redis,gitaly,postgresql}/log/config are missing.

I started version gitlab-ee:12.10.1-ee.0 and executed

docker exec <containername> cp /var/opt/gitlab/redis/redis.conf /opt/gitlab/sv/redis/log/config
docker exec <containername> cp /var/opt/gitlab/gitaly/config.toml /opt/gitlab/sv/gitaly/log/config
docker exec <containername> cp /var/opt/gitlab/postgresql/data/postgresql.conf /opt/gitlab/sv/postgresql/log/config

Then gitlab is starting again.
Afterwards I was able to upgrade to gitlab-ee:13.1.1-ee.0

6 Likes
8

Thanks this works!

9

Thx it worked also for me

10

It also worked for me. @conrad784 thank you!

11

Hi,
i’m facing the same problem. I tried to update from 12.2.1-ce to 13.1.4-ce.0 and it failed.
Now i can’t get it up and running again. Not even with the old image.
I’m using the gitlab-ce docker images and the container stops immediatly after the error.
Any suggestions how i can get the copy commands from the solutions in there?

12

I found an article [https://7thzero.com/blog/ad-hoc-repairs-to-a-failed-gitlab-ce-upgrade-12-8-greater-than-13-0-8](http://Ad-hoc repairs to a failed gitlab-ce upgrade (12.8 -> 13.0.8))
That seems to be helping with my problem.

1 Like
13

The article helped me…

My container didn´t start anymore.
I “simply” put empty “config” files in the folders like in the article (“simply” took me ~3 hours).

# touch config
# docker cp config f0a53fbfc5ff07:/opt/gitlab/sv/redis/log
# docker cp config f0a53fbfc5ff07:/var/log/gitlab/gitaly
# docker cp config f0a53fbfc5ff07:/opt/gitlab/sv/gitaly/log
# docker cp config f0a53fbfc5ff07:/opt/gitlab/sv/postgresql/log

Then the migrations ran successfully (version 12.10.1-ce.0) and migrated Postgres tables and so on.

But… I still have a problem: when I destroy the container and restart it, the problem with the missing config files reoccurs

14

Follow up on my last statement:

I migrated then from 12.10.1 to ‘gitlab/gitlab-ce:13.0.10-ce.0’ where I had to manually add the empty files again and THEN migrated to gitlab/gitlab-ce:13.2.0-ce.0 … wich worked without adding the files.

The container now works after destroying and recreation without manual input.

LEARNING:
Don´t use “:latest” tag in docker descriptors. Always use a specific version. Do backups before upgrading :expressionless: :no_mouth:

15

conrad784, thank you so much, my gitlab docker container is back up and running!

16

I needed to alter this slightly (as the directories were also missing):

docker-compose exec gitlab bash -c 'mkdir -p /opt/gitlab/sv/redis/log/ && cp /var/opt/gitlab/redis/redis.conf /opt/gitlab/sv/redis/log/config'
docker-compose exec gitlab bash -c 'mkdir -p /opt/gitlab/sv/gitaly/log/ && cp /var/opt/gitlab/gitaly/config.toml /opt/gitlab/sv/gitaly/log/config'
docker-compose exec gitlab bash -c 'mkdir -p /opt/gitlab/sv/postgresql/log/ && cp /var/opt/gitlab/postgresql/data/postgresql.conf /opt/gitlab/sv/postgresql/log/config'
17

You save my day ! Thank you :slight_smile: