Gilab-ce upgrade 14.8.4->14.9.0 problems

leasharon · June 17, 2022, 10:14am

Hello
I am trying to upgrade gitlab but am having some errors

can some body give me tips or some hulp to resolve this?
i run gitlab in a container

Thanks

grove · June 21, 2022, 8:21am

Don’t post pictures of text!

dnsmichi · June 21, 2022, 10:14am

@leasharon Welcome to our community

Please use the code tags or three backticks to format the output of your errors as text, so that we can copy and inspect it ourselves.

```
first log line
second log line
```

Please also share how you start the container, and which parameters are passed to the command. The file exists error could lead to a mounted volume or wrong paths involved.

With regards to the upgrade path, suggest verifying that you run the latest 14.8.x version first and then try an upgrade to 14.9.x.

GitLab Docker images | GitLab => Upgrading GitLab | GitLab

Note that 14.9.0 runs batched background migrations for the database which need to finish. Upgrading GitLab | GitLab

leasharon · June 24, 2022, 8:07am

Hi thanks

I start the container with these parameters

export GITLAB_HOME=/home/linux/srv
podman run -d  \
  --hostname redhatdev.mydomain.be \
  --publish 8081:443 --publish 8080:80 --publish 12022:22 \
  --name gitlab-ce \
  --restart always \
  --volume $GITLAB_HOME/config:/etc/gitlab:Z \
  --volume $GITLAB_HOME/logs:/var/log/gitlab:Z \
  --volume $GITLAB_HOME/data:/var/opt/gitlab:Z \
  --health-cmd='["curl", "--fail", "http://localhost"]' \
  gitlab/gitlab-ce:14.8.6-ce.0

I have tried to upgrade to different version 14.8.x but I always get the same errors.
I suspect there are some permissions issues, which I have tried to solve with

 podman exec -it gitlab-ce update-permissions
podman restart gitlab-ce

but this doesn’t solve my problem

Luckily I can always go back to 14.8.4 because I can restore a backup

the logs

Thank you for using GitLab Docker Image!
Current version: gitlab-ce=14.8.6-ce.0

.
.
.

================================================================================e[0m
      e[31mError executing action `create` on resource 'link[/var/log/gitlab/logrotate/config]'e[0m
      ================================================================================e[0m
      
e[0m      Errno::EEXISTe[0m
      -------------e[0m
      File exists @ syserr_fail2_in - /var/log/gitlab/logrotate/confige[0m
      
e[0m      Cookbook Trace:e[0m
      ---------------e[0m
      /opt/gitlab/embedded/cookbooks/cache/cookbooks/runit/libraries/provider_runit_service.rb:295:in `block in <class:RunitService>'e[0m
      
e[0m      Resource Declaration:e[0m
      ---------------------e[0m
      # In /opt/gitlab/embedded/cookbooks/cache/cookbooks/runit/libraries/provider_runit_service.rb
e[0m      
e[0m      144:             link ::File.join(new_resource.options[:log_directory], 'config') do
e[0m      145:               to ::File.join(sv_dir_name, 'log', 'config')
e[0m      146:             end
e[0m      147: 
e[0m      
e[0m      Compiled Resource:e[0m
      ------------------e[0m
      # Declared in /opt/gitlab/embedded/cookbooks/cache/cookbooks/runit/libraries/provider_runit_service.rb:144:in `block in <class:RunitService>'
e[0m      
e[0m      link("/var/log/gitlab/logrotate/config") do
e[0m        action [:create]
e[0m        default_guard_interpreter :default
e[0m        declared_type :link
e[0m        cookbook_name "logrotate"
e[0m        to "/opt/gitlab/sv/logrotate/log/config"
e[0m        target_file "/var/log/gitlab/logrotate/config"
e[0m      end
e[0m      
e[0m      System Info:e[0m
      ------------e[0m
      chef_version=15.17.4
e[0m      platform=ubuntu
e[0m      platform_version=20.04
e[0m      ruby=ruby 2.7.5p203 (2021-11-24 revision f69aeb8314) [x86_64-linux]
e[0m      program_name=/opt/gitlab/embedded/bin/chef-client
e[0m      executable=/opt/gitlab/embedded/bin/chef-cliente[0m
      
e[0m    * ruby_block[restart_service] action run (skipped due to only_if)
    * ruby_block[reload_log_service] action create
      * ruby_block[restart_service] action nothing (skipped due to action :nothing)
      * ruby_block[restart_log_service] action nothing (skipped due to action :nothing)
      * ruby_block[reload_log_service] action nothing (skipped due to action :nothing)
      * directory[/opt/gitlab/sv/logrotate] action create (up to date)
      * template[/opt/gitlab/sv/logrotate/run] action create (up to date)
      * directory[/opt/gitlab/sv/logrotate/log] action create (up to date)
      * directory[/opt/gitlab/sv/logrotate/log/main] action create (up to date)
      * template[/opt/gitlab/sv/logrotate/log/config] action create (up to date)
      * ruby_block[verify_chown_persisted_on_logrotate] action nothing (skipped due to action :nothing)
      * link[/var/log/gitlab/logrotate/config] action create
        e[0m
        ================================================================================e[0m
        e[31mError executing action `create` on resource 'link[/var/log/gitlab/logrotate/config]'e[0m
        ================================================================================e[0m
        
e[0m        Errno::EEXISTe[0m
        -------------e[0m
        File exists @ syserr_fail2_in - /var/log/gitlab/logrotate/confige[0m
        
e[0m        Cookbook Trace:e[0m
        ---------------e[0m
        /opt/gitlab/embedded/cookbooks/cache/cookbooks/runit/libraries/provider_runit_service.rb:295:in `block in <class:RunitService>'
e[0m        /opt/gitlab/embedded/cookbooks/cache/cookbooks/runit/libraries/provider_runit_service.rb:75:in `block (3 levels) in <class:RunitService>'
e[0m        /opt/gitlab/embedded/cookbooks/cache/cookbooks/runit/libraries/provider_runit_service.rb:295:in `block in <class:RunitService>'e[0m
        
e[0m        Resource Declaration:e[0m
        ---------------------e[0m
        # In /opt/gitlab/embedded/cookbooks/cache/cookbooks/runit/libraries/provider_runit_service.rb
e[0m        
e[0m        144:             link ::File.join(new_resource.options[:log_directory], 'config') do
e[0m        145:               to ::File.join(sv_dir_name, 'log', 'config')
e[0m        146:             end
e[0m        147: 
e[0m        
e[0m        Compiled Resource:e[0m
        ------------------e[0m
        # Declared in /opt/gitlab/embedded/cookbooks/cache/cookbooks/runit/libraries/provider_runit_service.rb:144:in `block in <class:RunitService>'
e[0m        
e[0m        link("/var/log/gitlab/logrotate/config") do
e[0m          action [:create]
e[0m          default_guard_interpreter :default
e[0m          declared_type :link
e[0m          cookbook_name "logrotate"
e[0m          to "/opt/gitlab/sv/logrotate/log/config"
e[0m          target_file "/var/log/gitlab/logrotate/config"
e[0m        end
e[0m        
e[0m        System Info:e[0m
        ------------e[0m
        chef_version=15.17.4
e[0m        platform=ubuntu
e[0m        platform_version=20.04
e[0m        ruby=ruby 2.7.5p203 (2021-11-24 revision f69aeb8314) [x86_64-linux]
e[0m        program_name=/opt/gitlab/embedded/bin/chef-client
e[0m        executable=/opt/gitlab/embedded/bin/chef-cliente[0m
        
e[0m      e[0m
      ================================================================================e[0m
      e[31mError executing action `create` on resource 'ruby_block[reload_log_service]'e[0m
      ================================================================================e[0m
      
e[0m      Errno::EEXISTe[0m
      -------------e[0m
      link[/var/log/gitlab/logrotate/config] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/runit/libraries/provider_runit_service.rb line 144) had an error: Errno::EEXIST: File exists @ syserr_fail2_in - /var/log/gitlab/logrotate/confige[0m
      
e[0m      Cookbook Trace:e[0m
      ---------------e[0m
      /opt/gitlab/embedded/cookbooks/cache/cookbooks/runit/libraries/provider_runit_service.rb:295:in `block in <class:RunitService>'
e[0m      /opt/gitlab/embedded/cookbooks/cache/cookbooks/runit/libraries/provider_runit_service.rb:75:in `block (3 levels) in <class:RunitService>'
e[0m      /opt/gitlab/embedded/cookbooks/cache/cookbooks/runit/libraries/provider_runit_service.rb:295:in `block in <class:RunitService>'e[0m
      
e[0m      Resource Declaration:e[0m
      ---------------------e[0m
      # In /opt/gitlab/embedded/cookbooks/cache/cookbooks/runit/libraries/provider_runit_service.rb
e[0m      
e[0m       73:         ruby_block 'reload_log_service' do
e[0m       74:           block do
e[0m       75:             action_enable
e[0m       76:             reload_log_service
e[0m       77:           end
e[0m       78:           action :nothing
e[0m       79:         end
e[0m       80: 
e[0m      
e[0m      Compiled Resource:e[0m
      ------------------e[0m
      # Declared in /opt/gitlab/embedded/cookbooks/cache/cookbooks/runit/libraries/provider_runit_service.rb:73:in `block in <class:RunitService>'
e[0m      
e[0m      ruby_block("reload_log_service") do
e[0m        action [:nothing]
e[0m        default_guard_interpreter :default
e[0m        declared_type :ruby_block
e[0m        cookbook_name "logrotate"
e[0m        block #<Proc:0x00005627fceb6d90 /opt/gitlab/embedded/cookbooks/cache/cookbooks/runit/libraries/provider_runit_service.rb:74>
e[0m        block_name "reload_log_service"
e[0m      end
e[0m      
e[0m      System Info:e[0m
      ------------e[0m
      chef_version=15.17.4
e[0m      platform=ubuntu
e[0m      platform_version=20.04
e[0m      ruby=ruby 2.7.5p203 (2021-11-24 revision f69aeb8314) [x86_64-linux]
e[0m      program_name=/opt/gitlab/embedded/bin/chef-client
e[0m      executable=/opt/gitlab/embedded/bin/chef-cliente[0m
      
e[0m    e[0m
    ================================================================================e[0m
    e[31mError executing action `enable` on resource 'runit_service[logrotate]'e[0m
    ================================================================================e[0m
    
e[0m    Chef::Exceptions::MultipleFailurese[0m
    ----------------------------------e[0m
    Multiple failures occurred:
e[0m    * Errno::EEXIST occurred in Chef Infra Client run: link[/var/log/gitlab/logrotate/config] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/runit/libraries/provider_runit_service.rb line 144) had an error: Errno::EEXIST: File exists @ syserr_fail2_in - /var/log/gitlab/logrotate/config
e[0m    * Errno::EEXIST occurred in delayed notification: ruby_block[reload_log_service] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/runit/libraries/provider_runit_service.rb line 73) had an error: Errno::EEXIST: link[/var/log/gitlab/logrotate/config] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/runit/libraries/provider_runit_service.rb line 144) had an error: Errno::EEXIST: File exists @ syserr_fail2_in - /var/log/gitlab/logrotate/config
e[0m    
e[0m    Cookbook Trace:e[0m
    ---------------e[0m
    /opt/gitlab/embedded/cookbooks/cache/cookbooks/runit/libraries/provider_runit_service.rb:295:in `block in <class:RunitService>'e[0m
    
e[0m    Resource Declaration:e[0m
    ---------------------e[0m
    # In /opt/gitlab/embedded/cookbooks/cache/cookbooks/logrotate/recipes/enable.rb
e[0m    
e[0m     19: runit_service "logrotate" do
e[0m     20:   start_down node['logrotate']['ha']
e[0m     21:   control ['t']
e[0m     22:   options({
e[0m     23:     log_directory: node['logrotate']['log_directory']
e[0m     24:   }.merge(params))
e[0m     25:   log_options node['gitlab']['logging'].to_hash.merge(node['logrotate'].to_hash)
e[0m     26: end
e[0m     27: 
e[0m    
e[0m    Compiled Resource:e[0m
    ------------------e[0m
    # Declared in /opt/gitlab/embedded/cookbooks/cache/cookbooks/logrotate/recipes/enable.rb:19:in `from_file'
e[0m    
e[0m    runit_service("logrotate") do
e[0m      action [:enable]
e[0m      updated true
e[0m      updated_by_last_action true
e[0m      default_guard_interpreter :default
e[0m      declared_type :runit_service
e[0m      cookbook_name "logrotate"
e[0m      recipe_name "enable"
e[0m      start_down false
e[0m      control ["t"]
e[0m      service_name "logrotate"
e[0m      options {:env_dir=>"/opt/gitlab/sv/logrotate/env", :log_directory=>"/var/log/gitlab/logrotate"}
e[0m      log_options {"svlogd_size"=>209715200, "svlogd_num"=>30, "svlogd_timeout"=>86400, "svlogd_filter"=>"gzip", "svlogd_udp"=>nil, "svlogd_prefix"=>nil, "udp_log_shipping_host"=>nil, "udp_log_shipping_hostname"=>nil, "udp_log_shipping_port"=>514, "logrotate_frequency"=>"daily", "logrotate_maxsize"=>nil, "logrotate_size"=>nil, "logrotate_rotate"=>30, "logrotate_compress"=>"compress", "logrotate_method"=>"copytruncate", "logrotate_postrotate"=>nil, "logrotate_dateformat"=>nil, "enable"=>true, "ha"=>false, "dir"=>"/var/opt/gitlab/logrotate", "log_directory"=>"/var/log/gitlab/logrotate", "services"=>["nginx", "puma", "gitlab-rails", "gitlab-shell", "gitlab-workhorse", "gitlab-pages", "gitlab-kas", "gitaly", "mailroom"], "pre_sleep"=>600, "post_sleep"=>3000}
e[0m      run_template_name "logrotate"
e[0m      log_template_name "logrotate"
e[0m      check_script_template_name "logrotate"
e[0m      finish_script_template_name "logrotate"
e[0m      control_template_names {"t"=>"logrotate"}
e[0m    end
e[0m    
e[0m    System Info:e[0m
    ------------e[0m
    chef_version=15.17.4
e[0m    platform=ubuntu
e[0m    platform_version=20.04
e[0m    ruby=ruby 2.7.5p203 (2021-11-24 revision f69aeb8314) [x86_64-linux]
e[0m    program_name=/opt/gitlab/embedded/bin/chef-client
e[0m    executable=/opt/gitlab/embedded/bin/chef-cliente[0m
    
e[0mRecipe: gitlab::gitlab-railse[0m
  * execute[clear the gitlab-rails cache] action run (skipped due to not_if)
e[0m
Running handlers:e[0m

Notes:
Found old initial root password file at /etc/gitlab/initial_root_password and deleted it.

Running handlers complete
e[0mChef Infra Client failed. 55 resources updated in 07 secondse[0m

Notes:
Found old initial root password file at /etc/gitlab/initial_root_password and deleted it.

Thank you for using GitLab Docker Image!
Current version: gitlab-ce=14.8.6-ce.0

Configure GitLab for your system by editing /etc/gitlab/gitlab.rb file
And restart this container to reload settings.
To do it use docker exec:

  docker exec -it gitlab editor /etc/gitlab/gitlab.rb
  docker restart gitlab

For a comprehensive list of configuration options please see the Omnibus GitLab readme
https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/README.md

If this container fails to start due to permission problems try to fix it by executing:

  docker exec -it gitlab update-permissions
  docker restart gitlab

leasharon · June 24, 2022, 10:09am

Hallo

I found the problem. Selinux prevented /opt/gitlab/embedded/bin/ruby from getting read access on the lnk_files

/var/log/gitlab/redis/config
/var/log/gitlab/puma/config
-/var/opt/gitlab/nginx/logs
and /var/log/gitlab/logrotate/config.

dnsmichi · June 24, 2022, 11:35am

Glad you found the problem, thanks for sharing your solution with us

Topic		Replies	Views
Upgrade Gitlab CE 13.12.15 -> 14.0.12 failing Upgrade docker	1	1199	April 24, 2022
Gitlab CE Upgrade inconvenient Upgrade docker	8	1653	June 1, 2023
Issue upgrading gitlab 9.5.5-ce.0 Upgrade	0	1049	September 21, 2017
Error updating GitLab CE from version 12.1.6 to the latest How to Use GitLab	4	3805	January 7, 2020
Gitlab CE upgrade plan How to Use GitLab	1	984	March 13, 2023

Gilab-ce upgrade 14.8.4->14.9.0 problems

Related topics