Database migration problem during Gitlab 15+ upgrade

I am trying to update from 14.10.5 to 15.0.5. I’ve gradually updated to 14.10.5; everything worked fine.

When tying to update to 15.0.[1-5], I am hitting the following error during db migration:

rake aborted!
StandardError: An error has occurred, all later migrations canceled:

Gitlab::Database::BackgroundMigration::BatchedMigrationRunner::FailedToFinalize
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/background_migration/batched_migration_runner.rb:76:in `finalize'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/background_migration/batched_migration_runner.rb:10:in `finalize'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/batched_background_migration_helpers.rb:139:in `finalize_batched_background_migration'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:962:in `ensure_batched_background_migration_is_finished'
/opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20220213103859_remove_integrations_type.rb:48:in `up'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/lock_retry_mixin.rb:36:in `ddl_transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:115:in `configure_database'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:95:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'

Caused by:
Gitlab::Database::BackgroundMigration::BatchedMigrationRunner::FailedToFinalize: Gitlab::Database::BackgroundMigration::BatchedMigrationRunner::FailedToFinalize
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/background_migration/batched_migration_runner.rb:76:in `finalize'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/background_migration/batched_migration_runner.rb:10:in `finalize'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/batched_background_migration_helpers.rb:139:in `finalize_batched_background_migration'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:962:in `ensure_batched_background_migration_is_finished'
/opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20220213103859_remove_integrations_type.rb:48:in `up'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/lock_retry_mixin.rb:36:in `ddl_transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:115:in `configure_database'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:95:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Tasks: TOP => db:migrate
(See full trace by running task with --trace)
== 20220213103859 RemoveIntegrationsType: migrating ===========================

Unfortunately, this error really doesn’t give me much info what exactly failed. I figured out that it’s the following migration that fails /opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20220213103859_remove_integrations_type.rb and prevents the other migrations to complete.

gitlab-rake db:migrate:redo VERSION=20220213103859 reproduces the error.

I tried various workarounds from the forums, most notably rails_migration[gitlab-rails] action run fails from 14.8.2 to 14.10.0 (Ubuntu 20.04) (#360377) · Issues · GitLab.org / GitLab · GitLab but none seem to apply/help.

Is anyone else encountering this? Any idea how I can get more info about what exactly fails during this migration?

System information
System:		Ubuntu 18.04
Proxy:		no
Current User:	git
Using RVM:	no
Ruby Version:	2.7.5p203
Gem Version:	3.1.4
Bundler Version:2.2.33
Rake Version:	13.0.6
Redis Version:	6.2.6
Sidekiq Version:6.4.0
Go Version:	unknown

GitLab information
Version:	14.10.5-ee
Revision:	af6735ad95f
Directory:	/opt/gitlab/embedded/service/gitlab-rails
DB Adapter:	PostgreSQL
DB Version:	12.7
URL:		https://gitlab.domain.com
HTTP Clone URL:	https://gitlab.domain.com/some-group/some-project.git
SSH Clone URL:	git@gitlab.domain.com:some-group/some-project.git
Elasticsearch:	no
Geo:		no
Using LDAP:	no
Using Omniauth:	no

GitLab Shell
Version:	13.25.1
Repository storage paths:
- default: 	/var/opt/gitlab/git-data/repositories
GitLab Shell path:		/opt/gitlab/embedded/service/gitlab-shell

I just did a helm upgrade to 15.1.1 from 15.0.1 and having the same issue.

Upgrading to Gitlab 15.04 (chart v6.0.4) works fine. But to GitLab 15.1.1 (chart v6.1.1) I get the following error in the migration pod

Writing /srv/gitlab/config/cable.yml
Writing /srv/gitlab/config/database.yml
Writing /srv/gitlab/config/gitlab.yml
Writing /srv/gitlab/config/resque.yml
Begin parsing .tpl templates from /var/opt/gitlab/templates
Copying other config files found in /var/opt/gitlab/templates to /srv/gitlab/config
Attempting to run '/scripts/wait-for-deps /scripts/db-migrate' as a main process
Checking database migrations are up-to-date
Performing migrations (this will initialized if needed)
rake aborted!
StandardError: An error has occurred, all later migrations canceled:

Gitlab::Database::BackgroundMigration::BatchedMigrationRunner::FailedToFinalize
/srv/gitlab/lib/gitlab/database/background_migration/batched_migration_runner.rb:79:in `finalize'
/srv/gitlab/lib/gitlab/database/background_migration/batched_migration_runner.rb:10:in `finalize'
/srv/gitlab/lib/gitlab/database/migrations/batched_background_migration_helpers.rb:160:in `block (2 levels) in finalize_batched_background_migration'
/srv/gitlab/lib/gitlab/database/query_analyzers/base.rb:25:in `with_suppressed'
/srv/gitlab/lib/gitlab/database/migrations/batched_background_migration_helpers.rb:159:in `block in finalize_batched_background_migration'
/srv/gitlab/lib/gitlab/database/migrations/reestablished_connection_stack.rb:21:in `with_restored_connection_stack'
/srv/gitlab/lib/gitlab/database/migrations/batched_background_migration_helpers.rb:158:in `finalize_batched_background_migration'
/srv/gitlab/lib/gitlab/database/migration_helpers.rb:967:in `ensure_batched_background_migration_is_finished'
/srv/gitlab/db/post_migrate/20220601040233_finalize_backfill_issue_search_data.rb:10:in `up'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.4.7/lib/active_record/migration.rb:870:in `public_send'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.4.7/lib/active_record/migration.rb:870:in `exec_migration'
/srv/gitlab/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb:33:in `block in exec_migration'
/srv/gitlab/lib/gitlab/database/query_analyzer.rb:37:in `within'
/srv/gitlab/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb:30:in `exec_migration'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.4.7/lib/active_record/migration.rb:851:in `block (2 levels) in migrate'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.4.7/lib/active_record/migration.rb:850:in `block in migrate'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.4.7/lib/active_record/connection_adapters/abstract/connection_pool.rb:462:in `with_connection'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.4.7/lib/active_record/migration.rb:849:in `migrate'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.4.7/lib/active_record/migration.rb:1037:in `migrate'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.4.7/lib/active_record/migration.rb:1329:in `block in execute_migration_in_transaction'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.4.7/lib/active_record/migration.rb:1382:in `ddl_transaction'
/srv/gitlab/lib/gitlab/database/migrations/lock_retry_mixin.rb:36:in `ddl_transaction'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.4.7/lib/active_record/migration.rb:1328:in `execute_migration_in_transaction'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.4.7/lib/active_record/migration.rb:1302:in `each'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.4.7/lib/active_record/migration.rb:1302:in `migrate_without_lock'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.4.7/lib/active_record/migration.rb:1251:in `block in migrate'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.4.7/lib/active_record/migration.rb:1401:in `block in with_advisory_lock'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.4.7/lib/active_record/migration.rb:1416:in `block in with_advisory_lock_connection'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.4.7/lib/active_record/connection_adapters/abstract/connection_pool.rb:462:in `with_connection'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.4.7/lib/active_record/migration.rb:1416:in `with_advisory_lock_connection'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.4.7/lib/active_record/migration.rb:1397:in `with_advisory_lock'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.4.7/lib/active_record/migration.rb:1251:in `migrate'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.4.7/lib/active_record/migration.rb:1086:in `up'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.4.7/lib/active_record/migration.rb:1061:in `migrate'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.4.7/lib/active_record/tasks/database_tasks.rb:237:in `migrate'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.4.7/lib/active_record/railties/databases.rake:92:in `block (3 levels) in <main>'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.4.7/lib/active_record/railties/databases.rake:90:in `each'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.4.7/lib/active_record/railties/databases.rake:90:in `block (2 levels) in <main>'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/sentry-ruby-core-5.1.1/lib/sentry/rake.rb:26:in `execute'
/srv/gitlab/lib/tasks/gitlab/db.rake:115:in `configure_database'
/srv/gitlab/lib/tasks/gitlab/db.rake:95:in `block (3 levels) in <main>'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/sentry-ruby-core-5.1.1/lib/sentry/rake.rb:26:in `execute'

Caused by:
Gitlab::Database::BackgroundMigration::BatchedMigrationRunner::FailedToFinalize: Gitlab::Database::BackgroundMigration::BatchedMigrationRunner::FailedToFinalize
/srv/gitlab/lib/gitlab/database/background_migration/batched_migration_runner.rb:79:in `finalize'
/srv/gitlab/lib/gitlab/database/background_migration/batched_migration_runner.rb:10:in `finalize'
/srv/gitlab/lib/gitlab/database/migrations/batched_background_migration_helpers.rb:160:in `block (2 levels) in finalize_batched_background_migration'
/srv/gitlab/lib/gitlab/database/query_analyzers/base.rb:25:in `with_suppressed'
/srv/gitlab/lib/gitlab/database/migrations/batched_background_migration_helpers.rb:159:in `block in finalize_batched_background_migration'
/srv/gitlab/lib/gitlab/database/migrations/reestablished_connection_stack.rb:21:in `with_restored_connection_stack'
/srv/gitlab/lib/gitlab/database/migrations/batched_background_migration_helpers.rb:158:in `finalize_batched_background_migration'
/srv/gitlab/lib/gitlab/database/migration_helpers.rb:967:in `ensure_batched_background_migration_is_finished'
/srv/gitlab/db/post_migrate/20220601040233_finalize_backfill_issue_search_data.rb:10:in `up'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.4.7/lib/active_record/migration.rb:870:in `public_send'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.4.7/lib/active_record/migration.rb:870:in `exec_migration'
/srv/gitlab/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb:33:in `block in exec_migration'
/srv/gitlab/lib/gitlab/database/query_analyzer.rb:37:in `within'
/srv/gitlab/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb:30:in `exec_migration'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.4.7/lib/active_record/migration.rb:851:in `block (2 levels) in migrate'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.4.7/lib/active_record/migration.rb:850:in `block in migrate'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.4.7/lib/active_record/connection_adapters/abstract/connection_pool.rb:462:in `with_connection'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.4.7/lib/active_record/migration.rb:849:in `migrate'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.4.7/lib/active_record/migration.rb:1037:in `migrate'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.4.7/lib/active_record/migration.rb:1329:in `block in execute_migration_in_transaction'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.4.7/lib/active_record/migration.rb:1382:in `ddl_transaction'
/srv/gitlab/lib/gitlab/database/migrations/lock_retry_mixin.rb:36:in `ddl_transaction'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.4.7/lib/active_record/migration.rb:1328:in `execute_migration_in_transaction'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.4.7/lib/active_record/migration.rb:1302:in `each'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.4.7/lib/active_record/migration.rb:1302:in `migrate_without_lock'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.4.7/lib/active_record/migration.rb:1251:in `block in migrate'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.4.7/lib/active_record/migration.rb:1401:in `block in with_advisory_lock'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.4.7/lib/active_record/migration.rb:1416:in `block in with_advisory_lock_connection'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.4.7/lib/active_record/connection_adapters/abstract/connection_pool.rb:462:in `with_connection'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.4.7/lib/active_record/migration.rb:1416:in `with_advisory_lock_connection'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.4.7/lib/active_record/migration.rb:1397:in `with_advisory_lock'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.4.7/lib/active_record/migration.rb:1251:in `migrate'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.4.7/lib/active_record/migration.rb:1086:in `up'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.4.7/lib/active_record/migration.rb:1061:in `migrate'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.4.7/lib/active_record/tasks/database_tasks.rb:237:in `migrate'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.4.7/lib/active_record/railties/databases.rake:92:in `block (3 levels) in <main>'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.4.7/lib/active_record/railties/databases.rake:90:in `each'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.4.7/lib/active_record/railties/databases.rake:90:in `block (2 levels) in <main>'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/sentry-ruby-core-5.1.1/lib/sentry/rake.rb:26:in `execute'
/srv/gitlab/lib/tasks/gitlab/db.rake:115:in `configure_database'
/srv/gitlab/lib/tasks/gitlab/db.rake:95:in `block (3 levels) in <main>'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/sentry-ruby-core-5.1.1/lib/sentry/rake.rb:26:in `execute'
Tasks: TOP => db:migrate
(See full trace by running task with --trace)
main: == 20220601040233 FinalizeBackfillIssueSearchData: migrating ==================
main: -- transaction_open?()
main:    -> 0.0000s

Looking at /admin/background_migrations there are errors like this:

NameError	uninitialized constant Gitlab::BackgroundMigration::CleanupOrphanedRoutes Did you mean? Gitlab::BackgroundMigration::CleanupOrphanedLfsObjectsProjects

NameError	uninitialized constant Gitlab::BackgroundMigration::BackfillProjectMemberNamespaceId Did you mean? Gitlab::BackgroundMigration::BackfillProjectSettings

How are we supposed to fix these errors?

I managed to resolve this, but I don’t really know how.

It appears that migration fails if there are 1) any background migrations still pending/failed or 2) a migration step hangs/is stuck.

I retried the failed background migrations in v14, and then tried to redo the stuck migrations one by one via gitlab-rake db:migrate:redo VERSION=20220412060931 Once the last stuck went through, a gitlab-rake db:migrate finished the others. A reboot might have helped me, who knows.

2 Likes

had the same problem :frowning:
backtraced it to: PG::UndefinedColumn: ERROR: column integrations.template does not exist LINE 1: …nfig_name:main*/ DELETE FROM “integrations” WHERE "integrati…

so i’ve recreated those…

gitlab-rake db:migrate:down VERSION=20220325050642

and then gitlab-ctl reconfigure finished without any issues

I was able to complete the upgrade by going to /admin/background_migrations and re-trying the failed migrations, and at the same time running gitlab-rake db:migrate from the gitlab-toolbox POD.
After that once sidekiq restarted this time it started successfully. And it picked up the background migrations and completed them.