Upgrade Gitlab 15.1.2 - Gitlab 15.2.0 fails

How to Use GitLab Upgrade
1

Hi,

this morning Gitlab was upgraded from 15.1.2 to 15.2.0 but failed to complete.
There were a lot of errors seen during the upgrade coming from some rake tasks.
Therefore I wanted to go back again to 15.1.2 but that doesn’t seem to be possible either.
It looks like the database migration step fails during the downgrade :cry:

rails_migration[gitlab-rails] (gitlab::database_migrations line 51) had an error: Mixlib::ShellOut::ShellCommandFailed: bash[migrate gitlab-rails database] (gitlab::database_migrations line 18) had an error: Mixlib::ShellOut::ShellCommandFailed: Command execution failed. STDOUT/STDERR suppressed for sensitive resource

Anyone else came across this (or something similar)?
Configuration is Debian 10 (buster) 64bit, running on a AMD Phenom™ II X4 840 Processor with 32GB of memory.

3 Likes
2

Maybe this information is also needed

Recipe: gitlab::database_migrations
  * ruby_block[check remote PG version] action nothing (skipped due to action :nothing)
  * rails_migration[gitlab-rails] action run
    * bash[migrate gitlab-rails database] action run
      
      ================================================================================
      Error executing action `run` on resource 'bash[migrate gitlab-rails database]'
      ================================================================================
      
      Mixlib::ShellOut::ShellCommandFailed
      ------------------------------------
      Command execution failed. STDOUT/STDERR suppressed for sensitive resource
      
      Cookbook Trace: (most recent call first)
      ----------------------------------------
      /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/resources/rails_migration.rb:18:in `block in class_from_file'
      
      Resource Declaration:
      ---------------------
      suppressed sensitive resource output
      
      Compiled Resource:
      ------------------
      suppressed sensitive resource output
      
      System Info:
      ------------
      chef_version=17.10.0
      platform=debian
      platform_version=10
      ruby=ruby 2.7.5p203 (2021-11-24 revision f69aeb8314) [x86_64-linux]
      program_name=/opt/gitlab/embedded/bin/cinc-client
      executable=/home/opt/gitlab/embedded/bin/cinc-client
      
    
    ================================================================================
    Error executing action `run` on resource 'rails_migration[gitlab-rails]'
    ================================================================================
    
    Mixlib::ShellOut::ShellCommandFailed
    ------------------------------------
    bash[migrate gitlab-rails database] (gitlab::database_migrations line 18) had an error: Mixlib::ShellOut::ShellCommandFailed: Command execution failed. STDOUT/STDERR suppressed for sensitive resource
    
    Cookbook Trace: (most recent call first)
    ----------------------------------------
    /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/resources/rails_migration.rb:18:in `block in class_from_file'
    
    Resource Declaration:
    ---------------------
    # In /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/database_migrations.rb
    
     51: rails_migration "gitlab-rails" do
     52:   rake_task 'gitlab:db:configure'
     53:   logfile_prefix 'gitlab-rails-db-migrate'
     54:   helper migration_helper
     55: 
     56:   environment env_variables
     57:   dependent_services dependent_services
     58:   notifies :run, "execute[clear the gitlab-rails cache]", :immediately
     59:   notifies :run, "ruby_block[check remote PG version]", :immediately
     60: 
     61:   only_if { migration_helper.attributes_node['auto_migrate'] }
     62: end
    
    Compiled Resource:
    ------------------
    # Declared in /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/database_migrations.rb:51:in `from_file'
    
    rails_migration("gitlab-rails") do
      action [:run]
      default_guard_interpreter :default
      declared_type :rails_migration
      cookbook_name "gitlab"
      recipe_name "database_migrations"
      rake_task "gitlab:db:configure"
      logfile_prefix "gitlab-rails-db-migrate"
      helper "*sensitive value suppressed*"
      environment "*sensitive value suppressed*"
      dependent_services ["runit_service[puma]", "sidekiq_service[sidekiq]"]
      only_if { #code block }
    end
    
    System Info:
    ------------
    chef_version=17.10.0
    platform=debian
    platform_version=10
    ruby=ruby 2.7.5p203 (2021-11-24 revision f69aeb8314) [x86_64-linux]
    program_name=/opt/gitlab/embedded/bin/cinc-client
    executable=/home/opt/gitlab/embedded/bin/cinc-client
    
[2022-07-23T11:05:59+02:00] INFO: Running queued delayed notifications before re-raising exception

Running handlers:
[2022-07-23T11:05:59+02:00] ERROR: Running exception handlers
There was an error running gitlab-ctl reconfigure:

rails_migration[gitlab-rails] (gitlab::database_migrations line 51) had an error: Mixlib::ShellOut::ShellCommandFailed: bash[migrate gitlab-rails database] (gitlab::database_migrations line 18) had an error: Mixlib::ShellOut::ShellCommandFailed: Command execution failed. STDOUT/STDERR suppressed for sensitive resource

Running handlers complete
[2022-07-23T11:05:59+02:00] ERROR: Exception handlers complete
Infra Phase failed. 2 resources updated in 31 seconds
[2022-07-23T11:05:59+02:00] FATAL: Stacktrace dumped to /opt/gitlab/embedded/cookbooks/cache/chef-stacktrace.out
[2022-07-23T11:05:59+02:00] FATAL: ---------------------------------------------------------------------------------------
[2022-07-23T11:05:59+02:00] FATAL: PLEASE PROVIDE THE CONTENTS OF THE stacktrace.out FILE (above) IF YOU FILE A BUG REPORT
[2022-07-23T11:05:59+02:00] FATAL: ---------------------------------------------------------------------------------------
[2022-07-23T11:05:59+02:00] FATAL: Mixlib::ShellOut::ShellCommandFailed: rails_migration[gitlab-rails] (gitlab::database_migrations line 51) had an error: Mixlib::ShellOut::ShellCommandFailed: bash[migrate gitlab-rails database] (gitlab::database_migrations line 18) had an error: Mixlib::ShellOut::ShellCommandFailed: Command execution failed. STDOUT/STDERR suppressed for sensitive resource
3

I get the exact same error after upping to 15.2. However, the stack is very different, since I’m running GitLab from the official Docker images.

Maybe this is related to the database version, at least I’m using a shared Postgres instance which is ahead of the dockerized one, currently 14.4.

@Elfmeterversemmler Which version of Postgres are you using?

4

It‘s PG 12.10 which is delivered with Gitlab

5

Yep, same issue. Even just pulling docker pull gitlab/gitlab-ee:latest on a fresh VM fails with that migration error.

6
7

Figured out my problem, however, it most likely won’t help you: The Redis instance is also shared and an update which occured around the same time forced Redis into protected mode. Thus unreachable from the GitLab container, the migration failed with he error mentioned above. Setting a requirepass and configuring it in GitLab’s docker-compose.yml fixed everything.

8

Hi,
I’ve the same error during migration Gitlab from 15.1.3 to 15.2. I don’t host Giltab on docker, but directly on Ubuntu (installation via APT packages). I checked gitlab-rake db:migrate --trace command, and I’ve below message:

PG::CheckViolation: ERROR:  new row for relation "batched_background_migrations" violates check constraint "check_max_value_in_range"
DETAIL:  Failing row contains (31, 2022-07-26 11:03:05.431357+00, 2022-07-26 11:03:05.431357+00, 65, 41, 50000, 1000, 120, 1, BackfillImportedIssueSearchData, PrimaryKeyBatchingStrategy, issues, id, [], 35, 100, null, 2022-07-26 11:03:05.42924+00, null, gitlab_main).

In my situation fix the error this workaround: rails_migration[gitlab-rails] Error executing action run from ee-15.1.3 to ee-15.2.0 (#368631) · Issues · GitLab.org / GitLab · GitLab

9

Looks like this will be fixed in v15.2.2 so I’m planning to wait until then to upgrade to v15.2 :relieved: