Kubernetes - Upgrade GitLab 14.10.5 to 15.2.2 failed

How to Use GitLab Upgrade
1

Hello,

I am trying to update my gitlab which is on my kubernetes cluster.

Here is my update path:

14.10.5-ce.0 → 15.0.5-ce.0 → 15.0.2-ce.0 → 15.1.0-ce.0 → 15.1.4-ce.0 → 15.2.0-ce.0 -->15.2.2-ce.0

As soon as I upgrade to 15.x, my gitlab drops to 500.

Error :

gitlab-rake db:migrate --trace
** Invoke db:migrate (first_time)
** Invoke db:load_config (first_time)
** Invoke environment (first_time)
** Execute environment
** Execute db:load_config
** Invoke gitlab:db:validate_config (first_time)
** Invoke environment 
** Execute gitlab:db:validate_config
** Invoke schema_version_check (first_time)
** Invoke environment 
** Execute schema_version_check
** Execute db:migrate
== 20220213103859 RemoveIntegrationsType: migrating ===========================
rake aborted!
StandardError: An error has occurred, all later migrations canceled:

Gitlab::Database::BackgroundMigration::BatchedMigrationRunner::FailedToFinalize
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/background_migration/batched_migration_runner.rb:76:in `finalize'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/background_migration/batched_migration_runner.rb:10:in `finalize'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/batched_background_migration_helpers.rb:139:in `finalize_batched_background_migration'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:962:in `ensure_batched_background_migration_is_finished'
/opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20220213103859_remove_integrations_type.rb:48:in `up'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.7/lib/active_record/migration.rb:870:in `public_send'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.7/lib/active_record/migration.rb:870:in `exec_migration'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.7/lib/active_record/migration.rb:851:in `block (2 levels) in migrate'
/opt/gitlab/embedded/lib/ruby/2.7.0/benchmark.rb:293:in `measure'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.7/lib/active_record/migration.rb:850:in `block in migrate'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.7/lib/active_record/connection_adapters/abstract/connection_pool.rb:462:in `with_connection'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.7/lib/active_record/migration.rb:849:in `migrate'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.7/lib/active_record/migration.rb:1037:in `migrate'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.7/lib/active_record/migration.rb:1329:in `block in execute_migration_in_transaction'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.7/lib/active_record/migration.rb:1382:in `ddl_transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/lock_retry_mixin.rb:36:in `ddl_transaction'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.7/lib/active_record/migration.rb:1328:in `execute_migration_in_transaction'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.7/lib/active_record/migration.rb:1302:in `each'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.7/lib/active_record/migration.rb:1302:in `migrate_without_lock'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.7/lib/active_record/migration.rb:1251:in `block in migrate'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.7/lib/active_record/migration.rb:1401:in `block in with_advisory_lock'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.7/lib/active_record/migration.rb:1416:in `block in with_advisory_lock_connection'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.7/lib/active_record/connection_adapters/abstract/connection_pool.rb:462:in `with_connection'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.7/lib/active_record/migration.rb:1416:in `with_advisory_lock_connection'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.7/lib/active_record/migration.rb:1397:in `with_advisory_lock'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.7/lib/active_record/migration.rb:1251:in `migrate'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.7/lib/active_record/migration.rb:1086:in `up'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.7/lib/active_record/migration.rb:1061:in `migrate'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.7/lib/active_record/tasks/database_tasks.rb:237:in `migrate'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.7/lib/active_record/railties/databases.rake:92:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.7/lib/active_record/railties/databases.rake:90:in `each'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.7/lib/active_record/railties/databases.rake:90:in `block (2 levels) in <top (required)>'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:281:in `block in execute'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:281:in `each'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:281:in `execute'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/sentry-ruby-core-5.1.1/lib/sentry/rake.rb:26:in `execute'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:219:in `block in invoke_with_call_chain'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:199:in `synchronize'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:199:in `invoke_with_call_chain'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:188:in `invoke'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:160:in `invoke_task'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:116:in `block (2 levels) in top_level'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:116:in `each'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:116:in `block in top_level'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:125:in `run_with_threads'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:110:in `top_level'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:83:in `block in run'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:186:in `standard_exception_handling'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:80:in `run'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/exe/rake:27:in `<top (required)>'
/opt/gitlab/embedded/bin/rake:23:in `load'
/opt/gitlab/embedded/bin/rake:23:in `<top (required)>'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/bundler-2.2.33/lib/bundler/cli/exec.rb:58:in `load'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/bundler-2.2.33/lib/bundler/cli/exec.rb:58:in `kernel_load'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/bundler-2.2.33/lib/bundler/cli/exec.rb:23:in `run'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/bundler-2.2.33/lib/bundler/cli.rb:479:in `exec'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/bundler-2.2.33/lib/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/bundler-2.2.33/lib/bundler/vendor/thor/lib/thor/invocation.rb:127:in `invoke_command'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/bundler-2.2.33/lib/bundler/vendor/thor/lib/thor.rb:392:in `dispatch'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/bundler-2.2.33/lib/bundler/cli.rb:31:in `dispatch'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/bundler-2.2.33/lib/bundler/vendor/thor/lib/thor/base.rb:485:in `start'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/bundler-2.2.33/lib/bundler/cli.rb:25:in `start'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/bundler-2.2.33/exe/bundle:49:in `block in <top (required)>'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/bundler-2.2.33/lib/bundler/friendly_errors.rb:103:in `with_friendly_errors'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/bundler-2.2.33/exe/bundle:37:in `<top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'

Caused by:
Gitlab::Database::BackgroundMigration::BatchedMigrationRunner::FailedToFinalize: Gitlab::Database::BackgroundMigration::BatchedMigrationRunner::FailedToFinalize
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/background_migration/batched_migration_runner.rb:76:in `finalize'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/background_migration/batched_migration_runner.rb:10:in `finalize'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/batched_background_migration_helpers.rb:139:in `finalize_batched_background_migration'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:962:in `ensure_batched_background_migration_is_finished'
/opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20220213103859_remove_integrations_type.rb:48:in `up'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.7/lib/active_record/migration.rb:870:in `public_send'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.7/lib/active_record/migration.rb:870:in `exec_migration'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.7/lib/active_record/migration.rb:851:in `block (2 levels) in migrate'
/opt/gitlab/embedded/lib/ruby/2.7.0/benchmark.rb:293:in `measure'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.7/lib/active_record/migration.rb:850:in `block in migrate'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.7/lib/active_record/connection_adapters/abstract/connection_pool.rb:462:in `with_connection'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.7/lib/active_record/migration.rb:849:in `migrate'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.7/lib/active_record/migration.rb:1037:in `migrate'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.7/lib/active_record/migration.rb:1329:in `block in execute_migration_in_transaction'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.7/lib/active_record/migration.rb:1382:in `ddl_transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/lock_retry_mixin.rb:36:in `ddl_transaction'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.7/lib/active_record/migration.rb:1328:in `execute_migration_in_transaction'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.7/lib/active_record/migration.rb:1302:in `each'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.7/lib/active_record/migration.rb:1302:in `migrate_without_lock'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.7/lib/active_record/migration.rb:1251:in `block in migrate'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.7/lib/active_record/migration.rb:1401:in `block in with_advisory_lock'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.7/lib/active_record/migration.rb:1416:in `block in with_advisory_lock_connection'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.7/lib/active_record/connection_adapters/abstract/connection_pool.rb:462:in `with_connection'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.7/lib/active_record/migration.rb:1416:in `with_advisory_lock_connection'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.7/lib/active_record/migration.rb:1397:in `with_advisory_lock'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.7/lib/active_record/migration.rb:1251:in `migrate'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.7/lib/active_record/migration.rb:1086:in `up'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.7/lib/active_record/migration.rb:1061:in `migrate'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.7/lib/active_record/tasks/database_tasks.rb:237:in `migrate'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.7/lib/active_record/railties/databases.rake:92:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.7/lib/active_record/railties/databases.rake:90:in `each'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.7/lib/active_record/railties/databases.rake:90:in `block (2 levels) in <top (required)>'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:281:in `block in execute'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:281:in `each'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:281:in `execute'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/sentry-ruby-core-5.1.1/lib/sentry/rake.rb:26:in `execute'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:219:in `block in invoke_with_call_chain'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:199:in `synchronize'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:199:in `invoke_with_call_chain'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:188:in `invoke'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:160:in `invoke_task'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:116:in `block (2 levels) in top_level'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:116:in `each'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:116:in `block in top_level'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:125:in `run_with_threads'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:110:in `top_level'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:83:in `block in run'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:186:in `standard_exception_handling'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:80:in `run'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/exe/rake:27:in `<top (required)>'
/opt/gitlab/embedded/bin/rake:23:in `load'
/opt/gitlab/embedded/bin/rake:23:in `<top (required)>'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/bundler-2.2.33/lib/bundler/cli/exec.rb:58:in `load'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/bundler-2.2.33/lib/bundler/cli/exec.rb:58:in `kernel_load'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/bundler-2.2.33/lib/bundler/cli/exec.rb:23:in `run'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/bundler-2.2.33/lib/bundler/cli.rb:479:in `exec'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/bundler-2.2.33/lib/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/bundler-2.2.33/lib/bundler/vendor/thor/lib/thor/invocation.rb:127:in `invoke_command'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/bundler-2.2.33/lib/bundler/vendor/thor/lib/thor.rb:392:in `dispatch'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/bundler-2.2.33/lib/bundler/cli.rb:31:in `dispatch'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/bundler-2.2.33/lib/bundler/vendor/thor/lib/thor/base.rb:485:in `start'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/bundler-2.2.33/lib/bundler/cli.rb:25:in `start'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/bundler-2.2.33/exe/bundle:49:in `block in <top (required)>'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/bundler-2.2.33/lib/bundler/friendly_errors.rb:103:in `with_friendly_errors'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/bundler-2.2.33/exe/bundle:37:in `<top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Tasks: TOP => db:migrate

External postgres : version psql (PostgreSQL) 13.5

On the gitlab configuration for postgres :

        gitlab_rails['auto_migrate'] = false
        gitlab_rails['db_adapter'] = "postgresql"
        gitlab_rails['db_encoding'] = "unicode"
        gitlab_rails['db_database'] = "gitlabhq_production"
        gitlab_rails['db_username'] = "gitlab"
        gitlab_rails['db_password'] = "EDITED"
        gitlab_rails['db_host'] = "tools-postgres-ha-pgpool.databases.svc.cluster.local"
        gitlab_rails['db_port'] = "5432"
2

I also have this problem if the rights have jumped:

Énumération des objets: 10, fait.
Décompte des objets: 100% (10/10), fait.
Compression par delta en utilisant jusqu'à 8 fils d'exécution
Compression des objets: 100% (6/6), fait.
Écriture des objets: 100% (6/6), 8.83 Kio | 8.83 Mio/s, fait.
Total 6 (delta 3), réutilisés 0 (delta 0), réutilisés du pack 0
remote: GitLab: 401 Unauthorized
To x.dev:x/infrastructures/scw-x-public.git
 ! [remote rejected] main -> main (pre-receive hook declined)

I’m an administrator with my user

3

^^^ is this correct? why did you go back to 15.0.2-ce.0 after going to 15.0.5 ?

the official docs have 15.0.2 as the latest 15.0.x series but docker hub has 15.0.5 as of 12 days ago (when 15.1.4 was pushed). So, AFAIK, the upgrade path should be 14.10.5 → 15.0.5 → 15.1.4 → 15.2.Z. Unless the docs are wrong [15.2 docs don’t mention paths for 15.2.Z], I’m still at 14.6.1. I’d be most concerned about the use of 15.0.2 after 15.0.5. But I’m speculating…

4

No matter the version from 15.x onwards, I only have problems :confused:

Either random 500 errors or you can’t push and delete branches anymore