I am trying to update from 14.10.5 to 15.0.5. I’ve gradually updated to 14.10.5; everything worked fine.
When tying to update to 15.0.[1-5], I am hitting the following error during db migration:
rake aborted!
StandardError: An error has occurred, all later migrations canceled:
Gitlab::Database::BackgroundMigration::BatchedMigrationRunner::FailedToFinalize
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/background_migration/batched_migration_runner.rb:76:in `finalize'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/background_migration/batched_migration_runner.rb:10:in `finalize'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/batched_background_migration_helpers.rb:139:in `finalize_batched_background_migration'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:962:in `ensure_batched_background_migration_is_finished'
/opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20220213103859_remove_integrations_type.rb:48:in `up'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/lock_retry_mixin.rb:36:in `ddl_transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:115:in `configure_database'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:95:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Caused by:
Gitlab::Database::BackgroundMigration::BatchedMigrationRunner::FailedToFinalize: Gitlab::Database::BackgroundMigration::BatchedMigrationRunner::FailedToFinalize
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/background_migration/batched_migration_runner.rb:76:in `finalize'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/background_migration/batched_migration_runner.rb:10:in `finalize'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/batched_background_migration_helpers.rb:139:in `finalize_batched_background_migration'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:962:in `ensure_batched_background_migration_is_finished'
/opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20220213103859_remove_integrations_type.rb:48:in `up'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/lock_retry_mixin.rb:36:in `ddl_transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:115:in `configure_database'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:95:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Tasks: TOP => db:migrate
(See full trace by running task with --trace)
== 20220213103859 RemoveIntegrationsType: migrating ===========================
Unfortunately, this error really doesn’t give me much info what exactly failed. I figured out that it’s the following migration that fails /opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20220213103859_remove_integrations_type.rb and prevents the other migrations to complete.
gitlab-rake db:migrate:redo VERSION=20220213103859 reproduces the error.
I tried various workarounds from the forums, most notably rails_migration[gitlab-rails] action run fails from 14.8.2 to 14.10.0 (Ubuntu 20.04) (#360377) · Issues · GitLab.org / GitLab · GitLab but none seem to apply/help.
Is anyone else encountering this? Any idea how I can get more info about what exactly fails during this migration?
System information
System: Ubuntu 18.04
Proxy: no
Current User: git
Using RVM: no
Ruby Version: 2.7.5p203
Gem Version: 3.1.4
Bundler Version:2.2.33
Rake Version: 13.0.6
Redis Version: 6.2.6
Sidekiq Version:6.4.0
Go Version: unknown
GitLab information
Version: 14.10.5-ee
Revision: af6735ad95f
Directory: /opt/gitlab/embedded/service/gitlab-rails
DB Adapter: PostgreSQL
DB Version: 12.7
URL: https://gitlab.domain.com
HTTP Clone URL: https://gitlab.domain.com/some-group/some-project.git
SSH Clone URL: git@gitlab.domain.com:some-group/some-project.git
Elasticsearch: no
Geo: no
Using LDAP: no
Using Omniauth: no
GitLab Shell
Version: 13.25.1
Repository storage paths:
- default: /var/opt/gitlab/git-data/repositories
GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell
I just did a helm upgrade to 15.1.1 from 15.0.1 and having the same issue.
Upgrading to Gitlab 15.04 (chart v6.0.4) works fine. But to GitLab 15.1.1 (chart v6.1.1) I get the following error in the migration pod
Writing /srv/gitlab/config/cable.yml
Writing /srv/gitlab/config/database.yml
Writing /srv/gitlab/config/gitlab.yml
Writing /srv/gitlab/config/resque.yml
Begin parsing .tpl templates from /var/opt/gitlab/templates
Copying other config files found in /var/opt/gitlab/templates to /srv/gitlab/config
Attempting to run '/scripts/wait-for-deps /scripts/db-migrate' as a main process
Checking database migrations are up-to-date
Performing migrations (this will initialized if needed)
rake aborted!
StandardError: An error has occurred, all later migrations canceled:
Gitlab::Database::BackgroundMigration::BatchedMigrationRunner::FailedToFinalize
/srv/gitlab/lib/gitlab/database/background_migration/batched_migration_runner.rb:79:in `finalize'
/srv/gitlab/lib/gitlab/database/background_migration/batched_migration_runner.rb:10:in `finalize'
/srv/gitlab/lib/gitlab/database/migrations/batched_background_migration_helpers.rb:160:in `block (2 levels) in finalize_batched_background_migration'
/srv/gitlab/lib/gitlab/database/query_analyzers/base.rb:25:in `with_suppressed'
/srv/gitlab/lib/gitlab/database/migrations/batched_background_migration_helpers.rb:159:in `block in finalize_batched_background_migration'
/srv/gitlab/lib/gitlab/database/migrations/reestablished_connection_stack.rb:21:in `with_restored_connection_stack'
/srv/gitlab/lib/gitlab/database/migrations/batched_background_migration_helpers.rb:158:in `finalize_batched_background_migration'
/srv/gitlab/lib/gitlab/database/migration_helpers.rb:967:in `ensure_batched_background_migration_is_finished'
/srv/gitlab/db/post_migrate/20220601040233_finalize_backfill_issue_search_data.rb:10:in `up'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.4.7/lib/active_record/migration.rb:870:in `public_send'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.4.7/lib/active_record/migration.rb:870:in `exec_migration'
/srv/gitlab/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb:33:in `block in exec_migration'
/srv/gitlab/lib/gitlab/database/query_analyzer.rb:37:in `within'
/srv/gitlab/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb:30:in `exec_migration'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.4.7/lib/active_record/migration.rb:851:in `block (2 levels) in migrate'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.4.7/lib/active_record/migration.rb:850:in `block in migrate'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.4.7/lib/active_record/connection_adapters/abstract/connection_pool.rb:462:in `with_connection'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.4.7/lib/active_record/migration.rb:849:in `migrate'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.4.7/lib/active_record/migration.rb:1037:in `migrate'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.4.7/lib/active_record/migration.rb:1329:in `block in execute_migration_in_transaction'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.4.7/lib/active_record/migration.rb:1382:in `ddl_transaction'
/srv/gitlab/lib/gitlab/database/migrations/lock_retry_mixin.rb:36:in `ddl_transaction'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.4.7/lib/active_record/migration.rb:1328:in `execute_migration_in_transaction'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.4.7/lib/active_record/migration.rb:1302:in `each'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.4.7/lib/active_record/migration.rb:1302:in `migrate_without_lock'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.4.7/lib/active_record/migration.rb:1251:in `block in migrate'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.4.7/lib/active_record/migration.rb:1401:in `block in with_advisory_lock'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.4.7/lib/active_record/migration.rb:1416:in `block in with_advisory_lock_connection'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.4.7/lib/active_record/connection_adapters/abstract/connection_pool.rb:462:in `with_connection'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.4.7/lib/active_record/migration.rb:1416:in `with_advisory_lock_connection'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.4.7/lib/active_record/migration.rb:1397:in `with_advisory_lock'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.4.7/lib/active_record/migration.rb:1251:in `migrate'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.4.7/lib/active_record/migration.rb:1086:in `up'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.4.7/lib/active_record/migration.rb:1061:in `migrate'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.4.7/lib/active_record/tasks/database_tasks.rb:237:in `migrate'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.4.7/lib/active_record/railties/databases.rake:92:in `block (3 levels) in <main>'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.4.7/lib/active_record/railties/databases.rake:90:in `each'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.4.7/lib/active_record/railties/databases.rake:90:in `block (2 levels) in <main>'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/sentry-ruby-core-5.1.1/lib/sentry/rake.rb:26:in `execute'
/srv/gitlab/lib/tasks/gitlab/db.rake:115:in `configure_database'
/srv/gitlab/lib/tasks/gitlab/db.rake:95:in `block (3 levels) in <main>'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/sentry-ruby-core-5.1.1/lib/sentry/rake.rb:26:in `execute'
Caused by:
Gitlab::Database::BackgroundMigration::BatchedMigrationRunner::FailedToFinalize: Gitlab::Database::BackgroundMigration::BatchedMigrationRunner::FailedToFinalize
/srv/gitlab/lib/gitlab/database/background_migration/batched_migration_runner.rb:79:in `finalize'
/srv/gitlab/lib/gitlab/database/background_migration/batched_migration_runner.rb:10:in `finalize'
/srv/gitlab/lib/gitlab/database/migrations/batched_background_migration_helpers.rb:160:in `block (2 levels) in finalize_batched_background_migration'
/srv/gitlab/lib/gitlab/database/query_analyzers/base.rb:25:in `with_suppressed'
/srv/gitlab/lib/gitlab/database/migrations/batched_background_migration_helpers.rb:159:in `block in finalize_batched_background_migration'
/srv/gitlab/lib/gitlab/database/migrations/reestablished_connection_stack.rb:21:in `with_restored_connection_stack'
/srv/gitlab/lib/gitlab/database/migrations/batched_background_migration_helpers.rb:158:in `finalize_batched_background_migration'
/srv/gitlab/lib/gitlab/database/migration_helpers.rb:967:in `ensure_batched_background_migration_is_finished'
/srv/gitlab/db/post_migrate/20220601040233_finalize_backfill_issue_search_data.rb:10:in `up'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.4.7/lib/active_record/migration.rb:870:in `public_send'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.4.7/lib/active_record/migration.rb:870:in `exec_migration'
/srv/gitlab/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb:33:in `block in exec_migration'
/srv/gitlab/lib/gitlab/database/query_analyzer.rb:37:in `within'
/srv/gitlab/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb:30:in `exec_migration'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.4.7/lib/active_record/migration.rb:851:in `block (2 levels) in migrate'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.4.7/lib/active_record/migration.rb:850:in `block in migrate'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.4.7/lib/active_record/connection_adapters/abstract/connection_pool.rb:462:in `with_connection'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.4.7/lib/active_record/migration.rb:849:in `migrate'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.4.7/lib/active_record/migration.rb:1037:in `migrate'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.4.7/lib/active_record/migration.rb:1329:in `block in execute_migration_in_transaction'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.4.7/lib/active_record/migration.rb:1382:in `ddl_transaction'
/srv/gitlab/lib/gitlab/database/migrations/lock_retry_mixin.rb:36:in `ddl_transaction'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.4.7/lib/active_record/migration.rb:1328:in `execute_migration_in_transaction'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.4.7/lib/active_record/migration.rb:1302:in `each'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.4.7/lib/active_record/migration.rb:1302:in `migrate_without_lock'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.4.7/lib/active_record/migration.rb:1251:in `block in migrate'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.4.7/lib/active_record/migration.rb:1401:in `block in with_advisory_lock'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.4.7/lib/active_record/migration.rb:1416:in `block in with_advisory_lock_connection'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.4.7/lib/active_record/connection_adapters/abstract/connection_pool.rb:462:in `with_connection'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.4.7/lib/active_record/migration.rb:1416:in `with_advisory_lock_connection'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.4.7/lib/active_record/migration.rb:1397:in `with_advisory_lock'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.4.7/lib/active_record/migration.rb:1251:in `migrate'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.4.7/lib/active_record/migration.rb:1086:in `up'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.4.7/lib/active_record/migration.rb:1061:in `migrate'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.4.7/lib/active_record/tasks/database_tasks.rb:237:in `migrate'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.4.7/lib/active_record/railties/databases.rake:92:in `block (3 levels) in <main>'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.4.7/lib/active_record/railties/databases.rake:90:in `each'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.4.7/lib/active_record/railties/databases.rake:90:in `block (2 levels) in <main>'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/sentry-ruby-core-5.1.1/lib/sentry/rake.rb:26:in `execute'
/srv/gitlab/lib/tasks/gitlab/db.rake:115:in `configure_database'
/srv/gitlab/lib/tasks/gitlab/db.rake:95:in `block (3 levels) in <main>'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/sentry-ruby-core-5.1.1/lib/sentry/rake.rb:26:in `execute'
Tasks: TOP => db:migrate
(See full trace by running task with --trace)
main: == 20220601040233 FinalizeBackfillIssueSearchData: migrating ==================
main: -- transaction_open?()
main: -> 0.0000s
Looking at /admin/background_migrations there are errors like this:
NameError uninitialized constant Gitlab::BackgroundMigration::CleanupOrphanedRoutes Did you mean? Gitlab::BackgroundMigration::CleanupOrphanedLfsObjectsProjects
NameError uninitialized constant Gitlab::BackgroundMigration::BackfillProjectMemberNamespaceId Did you mean? Gitlab::BackgroundMigration::BackfillProjectSettings
How are we supposed to fix these errors?
I managed to resolve this, but I don’t really know how.
It appears that migration fails if there are 1) any background migrations still pending/failed or 2) a migration step hangs/is stuck.
I retried the failed background migrations in v14, and then tried to redo the stuck migrations one by one via gitlab-rake db:migrate:redo VERSION=20220412060931 Once the last stuck went through, a gitlab-rake db:migrate finished the others. A reboot might have helped me, who knows.
2 Likes
had the same problem 
backtraced it to: PG::UndefinedColumn: ERROR: column integrations.template does not exist LINE 1: …nfig_name:main*/ DELETE FROM “integrations” WHERE "integrati…
so i’ve recreated those…
gitlab-rake db:migrate:down VERSION=20220325050642
and then gitlab-ctl reconfigure finished without any issues
I was able to complete the upgrade by going to /admin/background_migrations and re-trying the failed migrations, and at the same time running gitlab-rake db:migrate from the gitlab-toolbox POD.
After that once sidekiq restarted this time it started successfully. And it picked up the background migrations and completed them.
I faced the same problem as @artooro when upgrading from 15.0.3 to 15.1.0.
The migration
down 20220601040233 Finalize backfill issue search data
failed. After some digging, I found the solution that worked for me!
There is a much older migration:
up 20211026070408 Backfill issue search data
Somehow the failing migration tries to finalize the much older migration. Everything worked fine for me after redoing the old migration
gitlab-rake db:migrate:redo VERSION=20211026070408
Afterwards, the migration worked flawlessly:
gitlab-rake db:migrate
The error is already a little bit older… but maybe that is going to help someone.