Failing background database migrations - 14.1.x to 14.2.x

I am trying to update my GitLab server that is currently running 14.1.7 to 14.2.5 to then upgrade it to the latest version. But I am receiving the error message that is described in the documentation here.

I’ve tried everything I could find online and also created an issue on the GitLab repo that contains more details.

When running the following gitlab-psql query, I get back a table with 2 remaining background migrations:

select job_class_name, table_name, column_name, job_arguments from batched_background_migrations where status <> 3;

            job_class_name             |  table_name  | column_name |                      job_arguments
---------------------------------------+--------------+-------------+----------------------------------------------------------
 BackfillIntegrationsTypeNew           | integrations | id          | []
 CopyColumnUsingBackgroundMigrationJob | deployments  | id          | [["deployable_id"], ["deployable_id_convert_to_bigint"]]
(2 rows)

I’ve tried forcing the migrations to run using the following commands:

docker exec -it gitlab gitlab-rake gitlab:background_migrations:finalize[BackfillIntegrationsTypeNew,integrations,id,'[]'] --trace

docker exec -it gitlab gitlab-rake gitlab:background_migrations:finalize[CopyColumnUsingBackgroundMigrationJob,deployments,id,'[["deployable_id"]\, ["deployable_id_convert_to_bigint"]]'] --trace

But they both fail with the following stacktrace:

** Invoke gitlab:background_migrations:finalize (first_time)
** Invoke environment (first_time)
** Execute environment
** Execute gitlab:background_migrations:finalize
rake aborted!
Gitlab::Database::BackgroundMigration::BatchedMigrationRunner::FailedToFinalize: Gitlab::Database::BackgroundMigration::BatchedMigrationRunner::FailedToFinalize
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/background_migration/batched_migration_runner.rb:74:in `finalize'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/background_migration/batched_migration_runner.rb:10:in `finalize'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/background_migrations.rake:13:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.3/lib/rake/task.rb:279:in `block in execute'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.3/lib/rake/task.rb:279:in `each'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.3/lib/rake/task.rb:279:in `execute'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.3/lib/rake/task.rb:219:in `block in invoke_with_call_chain'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.3/lib/rake/task.rb:199:in `synchronize'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.3/lib/rake/task.rb:199:in `invoke_with_call_chain'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.3/lib/rake/task.rb:188:in `invoke'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.3/lib/rake/application.rb:160:in `invoke_task'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.3/lib/rake/application.rb:116:in `block (2 levels) in top_level'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.3/lib/rake/application.rb:116:in `each'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.3/lib/rake/application.rb:116:in `block in top_level'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.3/lib/rake/application.rb:125:in `run_with_threads'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.3/lib/rake/application.rb:110:in `top_level'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.3/lib/rake/application.rb:83:in `block in run'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.3/lib/rake/application.rb:186:in `standard_exception_handling'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.3/lib/rake/application.rb:80:in `run'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.3/exe/rake:27:in `<top (required)>'
/opt/gitlab/embedded/bin/rake:23:in `load'
/opt/gitlab/embedded/bin/rake:23:in `<top (required)>'
/opt/gitlab/embedded/lib/ruby/site_ruby/2.7.0/bundler/cli/exec.rb:63:in `load'
/opt/gitlab/embedded/lib/ruby/site_ruby/2.7.0/bundler/cli/exec.rb:63:in `kernel_load'
/opt/gitlab/embedded/lib/ruby/site_ruby/2.7.0/bundler/cli/exec.rb:28:in `run'
/opt/gitlab/embedded/lib/ruby/site_ruby/2.7.0/bundler/cli.rb:476:in `exec'
/opt/gitlab/embedded/lib/ruby/site_ruby/2.7.0/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
/opt/gitlab/embedded/lib/ruby/site_ruby/2.7.0/bundler/vendor/thor/lib/thor/invocation.rb:127:in `invoke_command'
/opt/gitlab/embedded/lib/ruby/site_ruby/2.7.0/bundler/vendor/thor/lib/thor.rb:399:in `dispatch'
/opt/gitlab/embedded/lib/ruby/site_ruby/2.7.0/bundler/cli.rb:30:in `dispatch'
/opt/gitlab/embedded/lib/ruby/site_ruby/2.7.0/bundler/vendor/thor/lib/thor/base.rb:476:in `start'
/opt/gitlab/embedded/lib/ruby/site_ruby/2.7.0/bundler/cli.rb:24:in `start'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/bundler-2.1.4/exe/bundle:46:in `block in <top (required)>'
/opt/gitlab/embedded/lib/ruby/site_ruby/2.7.0/bundler/friendly_errors.rb:123:in `with_friendly_errors'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/bundler-2.1.4/exe/bundle:34:in `<top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Tasks: TOP => gitlab:background_migrations:finalize

I’ve tried everything I could find, including trying to rollback to version 14.0.11, but I don’t have a proper backup of the database from back when it was running 14.0.x. Using the current database with version 14.0.11 results in the GitLab instance returning HTTP 500 errors. Additionally, I’ve continued using the instance since upgrading to 14.1.x.

I’m stuck and don’t know what to do. Does anyone have suggestions?

Note I am running GitLab in the official docker image: gitlab/gitlab-ce:14.1.7-ce.0

Still having the same issue.

I owe you @Zandor300, because you solved this issue elsewhere:

Failing DB Migration: BatchedMigrationRunner::FailedToFinalize deployable_id_convert_to_bigint (#340193) · Issues · GitLab.org / GitLab · GitLab and the subsequent note.

Helped me immensely.

@joel.mclean I am glad my documenting has helped solved your issue! For the longest time I thought I was the only one having this issue, probably due to my weird setup running Docker on Unraid.

Are you also running GitLab in Docker? Which OS is your ‘root OS’?

I’ve since moved away from Unraid and Docker for my GitLab server, my GitLab instance was running too slow on HDD’s on Unraid. It is now running inside a Ubuntu VM on TrueNAS with sata SSD’s and has never been so fast. (New server also received a CPU and RAM upgrade)

I think the issue was systemic to upgrades where the upgrade documentation didn’t completely match the lived experience for self-hosted instances.

My platform is Debian, running gitlab CE compiled from source; very different to your build. I’m running the upgrades to get it to a version where I can transition from Source to a package managed build.

Very specifically, the transition from v13 to v14 where batch_background_migrations becomes a new thing, but there aren’t yet the tools/functions to process them if they fail, is where I ran aground.

Our instance is probably not going to become a containerised platform; we’ve only got about 50GB of projects and 20 users, and only really 3 active projects, so a single server is more than adequate; the server is a VM on VMware (currently, we’re a data centre provider who are negatively affected by the Broadcomm Acquisition so we’ll be moving away from VMware pretty shortly) with 16GB ram and 6 vcpus, which is overkill for day-to-day operations, but I had to boost it because the upgrades from source compile using webpack for 25 minutes and chews up a tonne of memory and CPU; you don’t want it to fail.

I mostly replied to this post, because it was the top hit when I was searching for my issue - but the actual answer being buried deep in another issue was worth crossposting, just in case any one else is unlucky enough to have inherited such an out of date system that they must upgrade!