Migrate gitlab-rails database fails while upgrading to 14.10.2-ee

How to Use GitLab Upgrade
1

Hi,

i have some issue with upgrading self-hosted omnibus gitlab-ee from 14.9.4-ee > 14.10.2-ee on Ubuntu Server 20.04

sudo apt-get update
results in:

...
  * rails_migration[gitlab-rails] action run
    * bash[migrate gitlab-rails database] action run

      ================================================================================
      Error executing action `run` on resource 'bash[migrate gitlab-rails database]'
      ================================================================================

      Mixlib::ShellOut::ShellCommandFailed
      ------------------------------------
      Command execution failed. STDOUT/STDERR suppressed for sensitive resource

      Resource Declaration:
      ---------------------
      suppressed sensitive resource output

      Compiled Resource:
      ------------------
      suppressed sensitive resource output

      System Info:
      ------------
      chef_version=15.17.4
      platform=ubuntu
      platform_version=20.04
      ruby=ruby 2.7.5p203 (2021-11-24 revision f69aeb8314) [x86_64-linux]
      program_name=/opt/gitlab/embedded/bin/chef-client
      executable=/opt/gitlab/embedded/bin/chef-client


    ================================================================================
    Error executing action `run` on resource 'rails_migration[gitlab-rails]'
    ================================================================================

    Mixlib::ShellOut::ShellCommandFailed
    ------------------------------------
    bash[migrate gitlab-rails database] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/resources/rails_migration.rb line 16) had an error: Mixlib::ShellOut::ShellCommandFailed: Command execution failed. STDOUT/STDERR suppressed for sensitive resource

    Resource Declaration:
    ---------------------
    # In /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/database_migrations.rb

     51: rails_migration "gitlab-rails" do
     52:   rake_task 'gitlab:db:configure'
     53:   logfile_prefix 'gitlab-rails-db-migrate'
     54:   helper migration_helper
     55:
     56:   environment env_variables
     57:   dependent_services dependent_services
     58:   notifies :run, "execute[clear the gitlab-rails cache]", :immediately
     59:   notifies :run, "ruby_block[check remote PG version]", :immediately
     60:
     61:   only_if { migration_helper.attributes_node['auto_migrate'] }
     62: end

    Compiled Resource:
    ------------------
    # Declared in /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/database_migrations.rb:51:in `from_file'

    rails_migration("gitlab-rails") do
      action [:run]
      default_guard_interpreter :default
      declared_type :rails_migration
      cookbook_name "gitlab"
      recipe_name "database_migrations"
      rake_task "gitlab:db:configure"
      logfile_prefix "gitlab-rails-db-migrate"
      helper "*sensitive value suppressed*"
      environment "*sensitive value suppressed*"
      dependent_services []
      only_if { #code block }
    end

    System Info:
    ------------
    chef_version=15.17.4
    platform=ubuntu
    platform_version=20.04
    ruby=ruby 2.7.5p203 (2021-11-24 revision f69aeb8314) [x86_64-linux]
    program_name=/opt/gitlab/embedded/bin/chef-client
    executable=/opt/gitlab/embedded/bin/chef-client

Recipe: gitaly::enable
  * runit_service[gitaly] action hup
    - send hup to runit_service[gitaly]
Recipe: gitlab-kas::enable
  * runit_service[gitlab-kas] action restart (up to date)

Running handlers:
There was an error running gitlab-ctl reconfigure:

rails_migration[gitlab-rails] (gitlab::database_migrations line 51) had an error: Mixlib::ShellOut::ShellCommandFailed: bash[migrate gitlab-rails database] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/resources/rails_migration.rb line 16) had an error: Mixlib::ShellOut::ShellCommandFailed: Command execution failed. STDOUT/STDERR suppressed for sensitive resource

Running handlers complete
Chef Infra Client failed. 17 resources updated in 32 seconds
===
There was an error running gitlab-ctl reconfigure. Please check the output above for more
details.
===

dpkg: error processing package gitlab-ee (--configure):
 installed gitlab-ee package post-installation script subprocess returned error exit status 1
Errors were encountered while processing:
 gitlab-ee
E: Sub-process /usr/bin/dpkg returned an error code (1)

Data imported from into gitbab-ee 14.6.2, but upgrade to 14.10 2 fails (upgrade to 14.9.4 results OK)
(data comes from gitlab-ee ombibus 14.6.2 hosted on CentOS 8)

Do you have any ideas whats wrong?

2

INFO same issue on fresh new installation (no data / Ubuntu 20.04.4 LTS [Server]).
upgrade gitbab-ee 14.6.2 > 14.10.2 results same error

3

You can’t upgrade directly from 14.6.2 to 14.10.2. As per the docs: Upgrading GitLab | GitLab

you will have to go from 14.6.2 to 14.9.0 first, and only then you can go to 14.10.x.

Although some people have also suggested on the forum, your best way could also be to upgrade to each 14.x.0 release. So from 14.6.2 to 14.7.0, then 14.8.0 then 14.9.0 then 14.10.0. Of course, between each upgrade make sure to wait for the background migrations to finish (it’s explained in the documentation).

4

Thank you very much.

I used to upgrade Gitlab continuously every few weeks before CentoOS 8 was “killed”. I didn’t noticed there are any upgrade paths and background migrations.

Migration to another OS was too late and there was a large gap to jump over on my own.