Upgrade from 16.9.1 to 16.11.4 Fail

I have upgraded from version 16.7 to 16.11.

After several errors in which the forum has helped me and I paste the thread then gives an error when updating the bbdd but I do not see if it is a problem of postgres and I am not clear where to look log and what is happening.

I paste here the update and reconfiguration process.

Update:

Recipe: gitlab::database_migrations
  * ruby_block[check remote PG version] action nothing (skipped due to action :nothing)
  * rails_migration[gitlab-rails] action run[2024-06-13T16:51:44+02:00] WARN: gitlab-rails does not have a log_group or default logdir mode defined. Setting to 0700.

    * bash_hide_env[migrate gitlab-rails database] action run
      [execute] rake aborted!
                ArgumentError: Data consistency is already set
                /opt/gitlab/embedded/service/gitlab-rails/app/workers/concerns/worker_attributes.rb:96:in `data_consistency'
                /opt/gitlab/embedded/service/gitlab-rails/config/initializers/mailer_retries.rb:6:in `block in <top (required)>'
                /opt/gitlab/embedded/service/gitlab-rails/config/environment.rb:7:in `<top (required)>'
                <internal:/opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:37:in `require'
                <internal:/opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:37:in `require'
                /opt/gitlab/embedded/bin/bundle:25:in `load'
                /opt/gitlab/embedded/bin/bundle:25:in `<main>'
                Tasks: TOP => gitlab:db:configure => environment
                (See full trace by running task with --trace)
      
      ================================================================================
      Error executing action `run` on resource 'bash_hide_env[migrate gitlab-rails database]'
      ================================================================================
      
      Mixlib::ShellOut::ShellCommandFailed
      ------------------------------------
      Expected process to exit with [0], but received '1'
      ---- Begin output of "bash"  ----
      STDOUT: rake aborted!
      ArgumentError: Data consistency is already set
      /opt/gitlab/embedded/service/gitlab-rails/app/workers/concerns/worker_attributes.rb:96:in `data_consistency'
      /opt/gitlab/embedded/service/gitlab-rails/config/initializers/mailer_retries.rb:6:in `block in <top (required)>'
      /opt/gitlab/embedded/service/gitlab-rails/config/environment.rb:7:in `<top (required)>'
      <internal:/opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:37:in `require'
      <internal:/opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:37:in `require'
      /opt/gitlab/embedded/bin/bundle:25:in `load'
      /opt/gitlab/embedded/bin/bundle:25:in `<main>'
      Tasks: TOP => gitlab:db:configure => environment
      (See full trace by running task with --trace)
      STDERR: 
      ---- End output of "bash"  ----
      Ran "bash"  returned 1
      
      Cookbook Trace: (most recent call first)
      ----------------------------------------
      /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/resources/rails_migration.rb:20:in `block in class_from_file'
      
      Resource Declaration:
      ---------------------
      # In /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/resources/rails_migration.rb
      
       20:   bash_hide_env "migrate #{new_resource.name} database" do
       21:     code <<-EOH
       22:     set -e
       23:     log_file="#{logging_settings[:log_directory]}/#{new_resource.logfile_prefix}-$(date +%Y-%m-%d-%H-%M-%S).log"
       24:     umask 077
       25:     /opt/gitlab/bin/gitlab-rake #{new_resource.rake_task} 2>& 1 | tee ${log_file}
       26:     STATUS=${PIPESTATUS[0]}
       27:     chown #{account_helper.gitlab_user}:#{account_helper.gitlab_group} ${log_file}
       28:     echo $STATUS > #{new_resource.helper.db_migrate_status_file}
       29:     exit $STATUS
       30:     EOH
       31: 
       32:     environment new_resource.environment if new_resource.property_is_set?(:environment)
       33:     new_resource.dependent_services.each do |svc|
       34:       notifies :restart, svc, :immediately
       35:     end
       36: 
       37:     not_if { new_resource.helper.migrated? }
       38:     sensitive false
       39:   end
       40: end
      
      Compiled Resource:
      ------------------
      # Declared in /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/resources/rails_migration.rb:20:in `block in class_from_file'
      
      bash_hide_env("migrate gitlab-rails database") do
        action [:run]
        default_guard_interpreter :default
        interpreter "bash"
        declared_type :bash_hide_env
        cookbook_name "gitlab"
        recipe_name "database_migrations"
        code "    set -e\n    log_file=\"/var/log/gitlab/gitlab-rails/gitlab-rails-db-migrate-$(date +%Y-%m-%d-%H-%M-%S).log\"\n    umask 077\n    /opt/gitlab/bin/gitlab-rake gitlab:db:configure 2>& 1 | tee ${log_file}\n    STATUS=${PIPESTATUS[0]}\n    chown git:git ${log_file}\n    echo $STATUS > /var/opt/gitlab/gitlab-rails/upgrade-status/db-migrate-c6697a9cd49e1cfa45cc31b1ecdb90ae-83a2fa9fae0\n    exit $STATUS\n"
        environment "*sensitive value suppressed*"
        not_if { #code block }
      end
      
      System Info:
      ------------
      chef_version=18.3.0
      platform=ubuntu
      platform_version=22.04
      ruby=ruby 3.1.4p223 (2023-03-30 revision 957bb7cb81) [x86_64-linux]
      program_name=/opt/gitlab/embedded/bin/cinc-client
      executable=/opt/gitlab/embedded/bin/cinc-client
      
    
    ================================================================================
    Error executing action `run` on resource 'rails_migration[gitlab-rails]'
    ================================================================================
    
    Mixlib::ShellOut::ShellCommandFailed
    ------------------------------------
    bash_hide_env[migrate gitlab-rails database] (gitlab::database_migrations line 20) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
    ---- Begin output of "bash"  ----
    STDOUT: rake aborted!
    ArgumentError: Data consistency is already set
    /opt/gitlab/embedded/service/gitlab-rails/app/workers/concerns/worker_attributes.rb:96:in `data_consistency'
    /opt/gitlab/embedded/service/gitlab-rails/config/initializers/mailer_retries.rb:6:in `block in <top (required)>'
    /opt/gitlab/embedded/service/gitlab-rails/config/environment.rb:7:in `<top (required)>'
    <internal:/opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:37:in `require'
    <internal:/opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:37:in `require'
    /opt/gitlab/embedded/bin/bundle:25:in `load'
    /opt/gitlab/embedded/bin/bundle:25:in `<main>'
    Tasks: TOP => gitlab:db:configure => environment
    (See full trace by running task with --trace)
    STDERR: 
    ---- End output of "bash"  ----
    Ran "bash"  returned 1
    
    Cookbook Trace: (most recent call first)
    ----------------------------------------
    /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/resources/rails_migration.rb:20:in `block in class_from_file'
    
    Resource Declaration:
    ---------------------
    # In /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/database_migrations.rb
    
     51: rails_migration "gitlab-rails" do
     52:   rake_task 'gitlab:db:configure'
     53:   logfile_prefix 'gitlab-rails-db-migrate'
     54:   helper migration_helper
     55: 
     56:   environment env_variables
     57:   dependent_services dependent_services
     58:   notifies :run, "execute[clear the gitlab-rails cache]", :immediately
     59:   notifies :run, "ruby_block[check remote PG version]", :immediately
     60: 
     61:   only_if { migration_helper.attributes_node['auto_migrate'] }
     62: end
    
    Compiled Resource:
    ------------------
    # Declared in /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/database_migrations.rb:51:in `from_file'
    
    rails_migration("gitlab-rails") do
      action [:run]
      default_guard_interpreter :default
      declared_type :rails_migration
      cookbook_name "gitlab"
      recipe_name "database_migrations"
      rake_task "gitlab:db:configure"
      logfile_prefix "gitlab-rails-db-migrate"
      helper "*sensitive value suppressed*"
      environment "*sensitive value suppressed*"
      dependent_services ["runit_service[puma]", "sidekiq_service[sidekiq]"]
      only_if { #code block }
    end
    
    System Info:
    ------------
    chef_version=18.3.0
    platform=ubuntu
    platform_version=22.04
    ruby=ruby 3.1.4p223 (2023-03-30 revision 957bb7cb81) [x86_64-linux]
    program_name=/opt/gitlab/embedded/bin/cinc-client
    executable=/opt/gitlab/embedded/bin/cinc-client
    
[2024-06-13T16:52:00+02:00] INFO: Running queued delayed notifications before re-raising exception
[2024-06-13T16:52:00+02:00] INFO: templatesymlink[Create a gitlab.yml and create a symlink to Rails root] sending restart action to runit_service[puma] (delayed)
Recipe: gitlab::puma
  * runit_service[puma] action restart (up to date)
[2024-06-13T16:52:00+02:00] INFO: templatesymlink[Create a gitlab.yml and create a symlink to Rails root] sending restart action to sidekiq_service[sidekiq] (delayed)
Recipe: gitlab::sidekiq
  * sidekiq_service[sidekiq] action restart
    * service[sidekiq] action nothing (skipped due to action :nothing)
    * runit_service[sidekiq] action restart (up to date)
     (up to date)
[2024-06-13T16:52:05+02:00] INFO: templatesymlink[Create a gitlab.yml and create a symlink to Rails root] sending run action to execute[clear the gitlab-rails cache] (delayed)
Recipe: gitlab::gitlab-rails
  * execute[clear the gitlab-rails cache] action run
    [execute] rake aborted!
              ArgumentError: Data consistency is already set
              /opt/gitlab/embedded/service/gitlab-rails/app/workers/concerns/worker_attributes.rb:96:in `data_consistency'
              /opt/gitlab/embedded/service/gitlab-rails/config/initializers/mailer_retries.rb:6:in `block in <top (required)>'
              /opt/gitlab/embedded/service/gitlab-rails/config/environment.rb:7:in `<top (required)>'
              <internal:/opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:37:in `require'
              <internal:/opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:37:in `require'
              /opt/gitlab/embedded/bin/bundle:25:in `load'
              /opt/gitlab/embedded/bin/bundle:25:in `<main>'
              Tasks: TOP => cache:clear => cache:clear:redis => environment
              (See full trace by running task with --trace)
    
    ================================================================================
    Error executing action `run` on resource 'execute[clear the gitlab-rails cache]'
    ================================================================================
    
    Mixlib::ShellOut::ShellCommandFailed
    ------------------------------------
    Expected process to exit with [0], but received '1'
    ---- Begin output of /opt/gitlab/bin/gitlab-rake cache:clear ----
    STDOUT: 
    STDERR: rake aborted!
    ArgumentError: Data consistency is already set
    /opt/gitlab/embedded/service/gitlab-rails/app/workers/concerns/worker_attributes.rb:96:in `data_consistency'
    /opt/gitlab/embedded/service/gitlab-rails/config/initializers/mailer_retries.rb:6:in `block in <top (required)>'
    /opt/gitlab/embedded/service/gitlab-rails/config/environment.rb:7:in `<top (required)>'
    <internal:/opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:37:in `require'
    <internal:/opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:37:in `require'
    /opt/gitlab/embedded/bin/bundle:25:in `load'
    /opt/gitlab/embedded/bin/bundle:25:in `<main>'
    Tasks: TOP => cache:clear => cache:clear:redis => environment
    (See full trace by running task with --trace)
    ---- End output of /opt/gitlab/bin/gitlab-rake cache:clear ----
    Ran /opt/gitlab/bin/gitlab-rake cache:clear returned 1
    
    Resource Declaration:
    ---------------------
    # In /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/gitlab-rails.rb
    
    543: execute "clear the gitlab-rails cache" do
    544:   command "/opt/gitlab/bin/gitlab-rake cache:clear"
    545:   action :nothing
    546:   not_if { omnibus_helper.not_listening?('redis') || !node['gitlab']['gitlab_rails']['rake_cache_clear'] }
    547: end
    548: 
    
    Compiled Resource:
    ------------------
    # Declared in /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/gitlab-rails.rb:543:in `from_file'
    
    execute("clear the gitlab-rails cache") do
      action [:nothing]
      default_guard_interpreter :execute
      command "/opt/gitlab/bin/gitlab-rake cache:clear"
      declared_type :execute
      cookbook_name "gitlab"
      recipe_name "gitlab-rails"
      not_if { #code block }
    end
    
    System Info:
    ------------
    chef_version=18.3.0
    platform=ubuntu
    platform_version=22.04
    ruby=ruby 3.1.4p223 (2023-03-30 revision 957bb7cb81) [x86_64-linux]
    program_name=/opt/gitlab/embedded/bin/cinc-client
    executable=/opt/gitlab/embedded/bin/cinc-client
    

Running handlers:
[2024-06-13T16:52:21+02:00] ERROR: Running exception handlers
There was an error running gitlab-ctl reconfigure:

Multiple failures occurred:
* Mixlib::ShellOut::ShellCommandFailed occurred in Chef Infra Client run: rails_migration[gitlab-rails] (gitlab::database_migrations line 51) had an error: Mixlib::ShellOut::ShellCommandFailed: bash_hide_env[migrate gitlab-rails database] (gitlab::database_migrations line 20) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
---- Begin output of "bash"  ----
STDOUT: rake aborted!
ArgumentError: Data consistency is already set
/opt/gitlab/embedded/service/gitlab-rails/app/workers/concerns/worker_attributes.rb:96:in `data_consistency'
/opt/gitlab/embedded/service/gitlab-rails/config/initializers/mailer_retries.rb:6:in `block in <top (required)>'
/opt/gitlab/embedded/service/gitlab-rails/config/environment.rb:7:in `<top (required)>'
<internal:/opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:37:in `require'
<internal:/opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:37:in `require'
/opt/gitlab/embedded/bin/bundle:25:in `load'
/opt/gitlab/embedded/bin/bundle:25:in `<main>'
Tasks: TOP => gitlab:db:configure => environment
(See full trace by running task with --trace)
STDERR: 
---- End output of "bash"  ----
Ran "bash"  returned 1
* Mixlib::ShellOut::ShellCommandFailed occurred in delayed notification: execute[clear the gitlab-rails cache] (gitlab::gitlab-rails line 543) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
---- Begin output of /opt/gitlab/bin/gitlab-rake cache:clear ----
STDOUT: 
STDERR: rake aborted!
ArgumentError: Data consistency is already set
/opt/gitlab/embedded/service/gitlab-rails/app/workers/concerns/worker_attributes.rb:96:in `data_consistency'
/opt/gitlab/embedded/service/gitlab-rails/config/initializers/mailer_retries.rb:6:in `block in <top (required)>'
/opt/gitlab/embedded/service/gitlab-rails/config/environment.rb:7:in `<top (required)>'
<internal:/opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:37:in `require'
<internal:/opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:37:in `require'
/opt/gitlab/embedded/bin/bundle:25:in `load'
/opt/gitlab/embedded/bin/bundle:25:in `<main>'
Tasks: TOP => cache:clear => cache:clear:redis => environment
(See full trace by running task with --trace)
---- End output of /opt/gitlab/bin/gitlab-rake cache:clear ----
Ran /opt/gitlab/bin/gitlab-rake cache:clear returned 1

Running handlers complete
[2024-06-13T16:52:21+02:00] ERROR: Exception handlers complete
Infra Phase failed. 3 resources updated in 44 seconds
[2024-06-13T16:52:21+02:00] FATAL: Stacktrace dumped to /opt/gitlab/embedded/cookbooks/cache/chef-stacktrace.out
[2024-06-13T16:52:21+02:00] FATAL: ---------------------------------------------------------------------------------------
[2024-06-13T16:52:21+02:00] FATAL: PLEASE PROVIDE THE CONTENTS OF THE stacktrace.out FILE (above) IF YOU FILE A BUG REPORT
[2024-06-13T16:52:21+02:00] FATAL: ---------------------------------------------------------------------------------------
[2024-06-13T16:52:21+02:00] FATAL: Chef::Exceptions::MultipleFailures: Multiple failures occurred:
* Mixlib::ShellOut::ShellCommandFailed occurred in Chef Infra Client run: rails_migration[gitlab-rails] (gitlab::database_migrations line 51) had an error: Mixlib::ShellOut::ShellCommandFailed: bash_hide_env[migrate gitlab-rails database] (gitlab::database_migrations line 20) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
---- Begin output of "bash"  ----
STDOUT: rake aborted!
ArgumentError: Data consistency is already set
/opt/gitlab/embedded/service/gitlab-rails/app/workers/concerns/worker_attributes.rb:96:in `data_consistency'
/opt/gitlab/embedded/service/gitlab-rails/config/initializers/mailer_retries.rb:6:in `block in <top (required)>'
/opt/gitlab/embedded/service/gitlab-rails/config/environment.rb:7:in `<top (required)>'
<internal:/opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:37:in `require'
<internal:/opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:37:in `require'
/opt/gitlab/embedded/bin/bundle:25:in `load'
/opt/gitlab/embedded/bin/bundle:25:in `<main>'
Tasks: TOP => gitlab:db:configure => environment
(See full trace by running task with --trace)
STDERR: 
---- End output of "bash"  ----
Ran "bash"  returned 1
* Mixlib::ShellOut::ShellCommandFailed occurred in delayed notification: execute[clear the gitlab-rails cache] (gitlab::gitlab-rails line 543) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
---- Begin output of /opt/gitlab/bin/gitlab-rake cache:clear ----
STDOUT: 
STDERR: rake aborted!
ArgumentError: Data consistency is already set
/opt/gitlab/embedded/service/gitlab-rails/app/workers/concerns/worker_attributes.rb:96:in `data_consistency'
/opt/gitlab/embedded/service/gitlab-rails/config/initializers/mailer_retries.rb:6:in `block in <top (required)>'
/opt/gitlab/embedded/service/gitlab-rails/config/environment.rb:7:in `<top (required)>'
<internal:/opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:37:in `require'
<internal:/opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:37:in `require'
/opt/gitlab/embedded/bin/bundle:25:in `load'
/opt/gitlab/embedded/bin/bundle:25:in `<main>'
Tasks: TOP => cache:clear => cache:clear:redis => environment
(See full trace by running task with --trace)
---- End output of /opt/gitlab/bin/gitlab-rake cache:clear ----
Ran /opt/gitlab/bin/gitlab-rake cache:clear returned 1

Have you tried going back to an earlier version?

As I said in the other thread, if the reconfigure is no longer blocking apt, you can probably try using an earlier version in the same series:

apt install gitlab-ce=16.11.3-ce.0 ; apt-mark hold gitlab-ce

If all else fails, hopefully you have a backup (Back up and restore GitLab | GitLab), and decide to retry the upgrade procedure: Iā€™d suggest the following upgrade path instead, as an example:

apt install gitlab-ce=16.7.7-ce.0
apt install gitlab-ce=16.8.7-ce.0
apt install gitlab-ce=16.9.8-ce.0
apt install gitlab-ce=16.10.6-ce.0
apt install gitlab-ce=16.11.3-ce.0

Remember you should wait a couple of hours (until the background migrations finish) before jumping to another version, see: Background migrations and upgrades | GitLab

1 Like

Hello Silva:

Every time gitlab needs to be updated it is a nightmare.
I have the server in proxmox and LXC container.
I have restored copy to version 16.7 that I had, I will download copy in local and I will test the update.

Best regards and thanks for your help