Gilab-ce upgrade 14.8.4->14.9.0 problems

Hello
I am trying to upgrade gitlab but am having some errors


can some body give me tips or some hulp to resolve this?
i run gitlab in a container

Thanks

Don’t post pictures of text!

@leasharon Welcome to our community :slight_smile:

Please use the code tags or three backticks to format the output of your errors as text, so that we can copy and inspect it ourselves.

```
first log line
second log line
```

Please also share how you start the container, and which parameters are passed to the command. The file exists error could lead to a mounted volume or wrong paths involved.

With regards to the upgrade path, suggest verifying that you run the latest 14.8.x version first and then try an upgrade to 14.9.x.

GitLab Docker images | GitLab => Upgrading GitLab | GitLab

Note that 14.9.0 runs batched background migrations for the database which need to finish. Upgrading GitLab | GitLab

Hi thanks

I start the container with these parameters

export GITLAB_HOME=/home/linux/srv
podman run -d  \
  --hostname redhatdev.mydomain.be \
  --publish 8081:443 --publish 8080:80 --publish 12022:22 \
  --name gitlab-ce \
  --restart always \
  --volume $GITLAB_HOME/config:/etc/gitlab:Z \
  --volume $GITLAB_HOME/logs:/var/log/gitlab:Z \
  --volume $GITLAB_HOME/data:/var/opt/gitlab:Z \
  --health-cmd='["curl", "--fail", "http://localhost"]' \
  gitlab/gitlab-ce:14.8.6-ce.0

I have tried to upgrade to different version 14.8.x but I always get the same errors.
I suspect there are some permissions issues, which I have tried to solve with

 podman exec -it gitlab-ce update-permissions
podman restart gitlab-ce

but this doesn’t solve my problem

Luckily I can always go back to 14.8.4 because I can restore a backup

the logs

Thank you for using GitLab Docker Image!
Current version: gitlab-ce=14.8.6-ce.0

.
.
.

================================================================================e[0m
      e[31mError executing action `create` on resource 'link[/var/log/gitlab/logrotate/config]'e[0m
      ================================================================================e[0m
      
e[0m      Errno::EEXISTe[0m
      -------------e[0m
      File exists @ syserr_fail2_in - /var/log/gitlab/logrotate/confige[0m
      
e[0m      Cookbook Trace:e[0m
      ---------------e[0m
      /opt/gitlab/embedded/cookbooks/cache/cookbooks/runit/libraries/provider_runit_service.rb:295:in `block in <class:RunitService>'e[0m
      
e[0m      Resource Declaration:e[0m
      ---------------------e[0m
      # In /opt/gitlab/embedded/cookbooks/cache/cookbooks/runit/libraries/provider_runit_service.rb
e[0m      
e[0m      144:             link ::File.join(new_resource.options[:log_directory], 'config') do
e[0m      145:               to ::File.join(sv_dir_name, 'log', 'config')
e[0m      146:             end
e[0m      147: 
e[0m      
e[0m      Compiled Resource:e[0m
      ------------------e[0m
      # Declared in /opt/gitlab/embedded/cookbooks/cache/cookbooks/runit/libraries/provider_runit_service.rb:144:in `block in <class:RunitService>'
e[0m      
e[0m      link("/var/log/gitlab/logrotate/config") do
e[0m        action [:create]
e[0m        default_guard_interpreter :default
e[0m        declared_type :link
e[0m        cookbook_name "logrotate"
e[0m        to "/opt/gitlab/sv/logrotate/log/config"
e[0m        target_file "/var/log/gitlab/logrotate/config"
e[0m      end
e[0m      
e[0m      System Info:e[0m
      ------------e[0m
      chef_version=15.17.4
e[0m      platform=ubuntu
e[0m      platform_version=20.04
e[0m      ruby=ruby 2.7.5p203 (2021-11-24 revision f69aeb8314) [x86_64-linux]
e[0m      program_name=/opt/gitlab/embedded/bin/chef-client
e[0m      executable=/opt/gitlab/embedded/bin/chef-cliente[0m
      
e[0m    * ruby_block[restart_service] action run (skipped due to only_if)
    * ruby_block[reload_log_service] action create
      * ruby_block[restart_service] action nothing (skipped due to action :nothing)
      * ruby_block[restart_log_service] action nothing (skipped due to action :nothing)
      * ruby_block[reload_log_service] action nothing (skipped due to action :nothing)
      * directory[/opt/gitlab/sv/logrotate] action create (up to date)
      * template[/opt/gitlab/sv/logrotate/run] action create (up to date)
      * directory[/opt/gitlab/sv/logrotate/log] action create (up to date)
      * directory[/opt/gitlab/sv/logrotate/log/main] action create (up to date)
      * template[/opt/gitlab/sv/logrotate/log/config] action create (up to date)
      * ruby_block[verify_chown_persisted_on_logrotate] action nothing (skipped due to action :nothing)
      * link[/var/log/gitlab/logrotate/config] action create
        e[0m
        ================================================================================e[0m
        e[31mError executing action `create` on resource 'link[/var/log/gitlab/logrotate/config]'e[0m
        ================================================================================e[0m
        
e[0m        Errno::EEXISTe[0m
        -------------e[0m
        File exists @ syserr_fail2_in - /var/log/gitlab/logrotate/confige[0m
        
e[0m        Cookbook Trace:e[0m
        ---------------e[0m
        /opt/gitlab/embedded/cookbooks/cache/cookbooks/runit/libraries/provider_runit_service.rb:295:in `block in <class:RunitService>'
e[0m        /opt/gitlab/embedded/cookbooks/cache/cookbooks/runit/libraries/provider_runit_service.rb:75:in `block (3 levels) in <class:RunitService>'
e[0m        /opt/gitlab/embedded/cookbooks/cache/cookbooks/runit/libraries/provider_runit_service.rb:295:in `block in <class:RunitService>'e[0m
        
e[0m        Resource Declaration:e[0m
        ---------------------e[0m
        # In /opt/gitlab/embedded/cookbooks/cache/cookbooks/runit/libraries/provider_runit_service.rb
e[0m        
e[0m        144:             link ::File.join(new_resource.options[:log_directory], 'config') do
e[0m        145:               to ::File.join(sv_dir_name, 'log', 'config')
e[0m        146:             end
e[0m        147: 
e[0m        
e[0m        Compiled Resource:e[0m
        ------------------e[0m
        # Declared in /opt/gitlab/embedded/cookbooks/cache/cookbooks/runit/libraries/provider_runit_service.rb:144:in `block in <class:RunitService>'
e[0m        
e[0m        link("/var/log/gitlab/logrotate/config") do
e[0m          action [:create]
e[0m          default_guard_interpreter :default
e[0m          declared_type :link
e[0m          cookbook_name "logrotate"
e[0m          to "/opt/gitlab/sv/logrotate/log/config"
e[0m          target_file "/var/log/gitlab/logrotate/config"
e[0m        end
e[0m        
e[0m        System Info:e[0m
        ------------e[0m
        chef_version=15.17.4
e[0m        platform=ubuntu
e[0m        platform_version=20.04
e[0m        ruby=ruby 2.7.5p203 (2021-11-24 revision f69aeb8314) [x86_64-linux]
e[0m        program_name=/opt/gitlab/embedded/bin/chef-client
e[0m        executable=/opt/gitlab/embedded/bin/chef-cliente[0m
        
e[0m      e[0m
      ================================================================================e[0m
      e[31mError executing action `create` on resource 'ruby_block[reload_log_service]'e[0m
      ================================================================================e[0m
      
e[0m      Errno::EEXISTe[0m
      -------------e[0m
      link[/var/log/gitlab/logrotate/config] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/runit/libraries/provider_runit_service.rb line 144) had an error: Errno::EEXIST: File exists @ syserr_fail2_in - /var/log/gitlab/logrotate/confige[0m
      
e[0m      Cookbook Trace:e[0m
      ---------------e[0m
      /opt/gitlab/embedded/cookbooks/cache/cookbooks/runit/libraries/provider_runit_service.rb:295:in `block in <class:RunitService>'
e[0m      /opt/gitlab/embedded/cookbooks/cache/cookbooks/runit/libraries/provider_runit_service.rb:75:in `block (3 levels) in <class:RunitService>'
e[0m      /opt/gitlab/embedded/cookbooks/cache/cookbooks/runit/libraries/provider_runit_service.rb:295:in `block in <class:RunitService>'e[0m
      
e[0m      Resource Declaration:e[0m
      ---------------------e[0m
      # In /opt/gitlab/embedded/cookbooks/cache/cookbooks/runit/libraries/provider_runit_service.rb
e[0m      
e[0m       73:         ruby_block 'reload_log_service' do
e[0m       74:           block do
e[0m       75:             action_enable
e[0m       76:             reload_log_service
e[0m       77:           end
e[0m       78:           action :nothing
e[0m       79:         end
e[0m       80: 
e[0m      
e[0m      Compiled Resource:e[0m
      ------------------e[0m
      # Declared in /opt/gitlab/embedded/cookbooks/cache/cookbooks/runit/libraries/provider_runit_service.rb:73:in `block in <class:RunitService>'
e[0m      
e[0m      ruby_block("reload_log_service") do
e[0m        action [:nothing]
e[0m        default_guard_interpreter :default
e[0m        declared_type :ruby_block
e[0m        cookbook_name "logrotate"
e[0m        block #<Proc:0x00005627fceb6d90 /opt/gitlab/embedded/cookbooks/cache/cookbooks/runit/libraries/provider_runit_service.rb:74>
e[0m        block_name "reload_log_service"
e[0m      end
e[0m      
e[0m      System Info:e[0m
      ------------e[0m
      chef_version=15.17.4
e[0m      platform=ubuntu
e[0m      platform_version=20.04
e[0m      ruby=ruby 2.7.5p203 (2021-11-24 revision f69aeb8314) [x86_64-linux]
e[0m      program_name=/opt/gitlab/embedded/bin/chef-client
e[0m      executable=/opt/gitlab/embedded/bin/chef-cliente[0m
      
e[0m    e[0m
    ================================================================================e[0m
    e[31mError executing action `enable` on resource 'runit_service[logrotate]'e[0m
    ================================================================================e[0m
    
e[0m    Chef::Exceptions::MultipleFailurese[0m
    ----------------------------------e[0m
    Multiple failures occurred:
e[0m    * Errno::EEXIST occurred in Chef Infra Client run: link[/var/log/gitlab/logrotate/config] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/runit/libraries/provider_runit_service.rb line 144) had an error: Errno::EEXIST: File exists @ syserr_fail2_in - /var/log/gitlab/logrotate/config
e[0m    * Errno::EEXIST occurred in delayed notification: ruby_block[reload_log_service] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/runit/libraries/provider_runit_service.rb line 73) had an error: Errno::EEXIST: link[/var/log/gitlab/logrotate/config] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/runit/libraries/provider_runit_service.rb line 144) had an error: Errno::EEXIST: File exists @ syserr_fail2_in - /var/log/gitlab/logrotate/config
e[0m    
e[0m    Cookbook Trace:e[0m
    ---------------e[0m
    /opt/gitlab/embedded/cookbooks/cache/cookbooks/runit/libraries/provider_runit_service.rb:295:in `block in <class:RunitService>'e[0m
    
e[0m    Resource Declaration:e[0m
    ---------------------e[0m
    # In /opt/gitlab/embedded/cookbooks/cache/cookbooks/logrotate/recipes/enable.rb
e[0m    
e[0m     19: runit_service "logrotate" do
e[0m     20:   start_down node['logrotate']['ha']
e[0m     21:   control ['t']
e[0m     22:   options({
e[0m     23:     log_directory: node['logrotate']['log_directory']
e[0m     24:   }.merge(params))
e[0m     25:   log_options node['gitlab']['logging'].to_hash.merge(node['logrotate'].to_hash)
e[0m     26: end
e[0m     27: 
e[0m    
e[0m    Compiled Resource:e[0m
    ------------------e[0m
    # Declared in /opt/gitlab/embedded/cookbooks/cache/cookbooks/logrotate/recipes/enable.rb:19:in `from_file'
e[0m    
e[0m    runit_service("logrotate") do
e[0m      action [:enable]
e[0m      updated true
e[0m      updated_by_last_action true
e[0m      default_guard_interpreter :default
e[0m      declared_type :runit_service
e[0m      cookbook_name "logrotate"
e[0m      recipe_name "enable"
e[0m      start_down false
e[0m      control ["t"]
e[0m      service_name "logrotate"
e[0m      options {:env_dir=>"/opt/gitlab/sv/logrotate/env", :log_directory=>"/var/log/gitlab/logrotate"}
e[0m      log_options {"svlogd_size"=>209715200, "svlogd_num"=>30, "svlogd_timeout"=>86400, "svlogd_filter"=>"gzip", "svlogd_udp"=>nil, "svlogd_prefix"=>nil, "udp_log_shipping_host"=>nil, "udp_log_shipping_hostname"=>nil, "udp_log_shipping_port"=>514, "logrotate_frequency"=>"daily", "logrotate_maxsize"=>nil, "logrotate_size"=>nil, "logrotate_rotate"=>30, "logrotate_compress"=>"compress", "logrotate_method"=>"copytruncate", "logrotate_postrotate"=>nil, "logrotate_dateformat"=>nil, "enable"=>true, "ha"=>false, "dir"=>"/var/opt/gitlab/logrotate", "log_directory"=>"/var/log/gitlab/logrotate", "services"=>["nginx", "puma", "gitlab-rails", "gitlab-shell", "gitlab-workhorse", "gitlab-pages", "gitlab-kas", "gitaly", "mailroom"], "pre_sleep"=>600, "post_sleep"=>3000}
e[0m      run_template_name "logrotate"
e[0m      log_template_name "logrotate"
e[0m      check_script_template_name "logrotate"
e[0m      finish_script_template_name "logrotate"
e[0m      control_template_names {"t"=>"logrotate"}
e[0m    end
e[0m    
e[0m    System Info:e[0m
    ------------e[0m
    chef_version=15.17.4
e[0m    platform=ubuntu
e[0m    platform_version=20.04
e[0m    ruby=ruby 2.7.5p203 (2021-11-24 revision f69aeb8314) [x86_64-linux]
e[0m    program_name=/opt/gitlab/embedded/bin/chef-client
e[0m    executable=/opt/gitlab/embedded/bin/chef-cliente[0m
    
e[0mRecipe: gitlab::gitlab-railse[0m
  * execute[clear the gitlab-rails cache] action run (skipped due to not_if)
e[0m
Running handlers:e[0m

Notes:
Found old initial root password file at /etc/gitlab/initial_root_password and deleted it.

Running handlers complete
e[0mChef Infra Client failed. 55 resources updated in 07 secondse[0m

Notes:
Found old initial root password file at /etc/gitlab/initial_root_password and deleted it.

Thank you for using GitLab Docker Image!
Current version: gitlab-ce=14.8.6-ce.0

Configure GitLab for your system by editing /etc/gitlab/gitlab.rb file
And restart this container to reload settings.
To do it use docker exec:

  docker exec -it gitlab editor /etc/gitlab/gitlab.rb
  docker restart gitlab

For a comprehensive list of configuration options please see the Omnibus GitLab readme
https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/README.md

If this container fails to start due to permission problems try to fix it by executing:

  docker exec -it gitlab update-permissions
  docker restart gitlab

Hallo

I found the problem. Selinux prevented /opt/gitlab/embedded/bin/ruby from getting read access on the lnk_files

  • /var/log/gitlab/redis/config
  • /var/log/gitlab/puma/config
    -/var/opt/gitlab/nginx/logs
    and /var/log/gitlab/logrotate/config.
1 Like

Glad you found the problem, thanks for sharing your solution with us :slight_smile: