(self-hosted) upgrade to gitlab 14.0.6: Background Migrations hung

We’re upgrading from 12.7 to 14.2 via Omnibus package installation (CentOS x86_64). I upgraded to 12.10.14, 13.0.14, 13.1.11, 13.12.8, and now 14.0.6. At each step, gitlab reconfigure was run, and all background jobs/migrations had completed before proceding. We are stuck now.

Background migrations page https://gitlab-test.local/admin/background_migrations shows 11 migration tasks all at 0.00% and status Active. These include items such as:

• CopyColumnUsingBackgroundMigrationJob: ci_build_trace_sections
• CopyColumnUsingBackgroundMigrationJob: ci_builds_metadata

There are 4 that have finished. There are no jobs in the “busy” queue under monitoring/background jobs.

The Troubleshooting section of the omnibus installer does not help here.
I tried the 14.1 docs to no avail (See below).

What can I do to facilitate the execution of these jobs?

 Gitlab::Database::BackgroundMigration::BatchedMigration.pluck(:id, :table_name, :status)
=> [[1, "ci_build_trace_sections", "active"], [2, "ci_builds_metadata", "active"], [3, "events", "active"], [4, "push_event_payloads", "active"], [5, "ci_job_artifacts", "active"], [6, "ci_sources_pipelines", "finished"], [7, "ci_build_needs", "active"], [8, "ci_builds", "active"], [9, "ci_builds_runner_session", "finished"], [10, "ci_build_trace_chunks", "finished"], [11, "taggings", "active"], [12, "deployments", "active"], [13, "geo_job_artifact_deleted_events", "finished"], [14, "ci_stages", "active"], [15, "ci_builds_metadata", "active"]]

sushi:[gitlab-test]/etc/gitlab #  gitlab-rake gitlab:background_migrations:finalize[CopyColumnUsingBackgroundMigrationJob,push_event_payloads,event_id,'[["event_id"]\, ["event_id_convert_to_bigint"]]']
rake aborted!
Don't know how to build task 'gitlab:background_migrations:finalize' (See the list of available tasks with `rake --tasks`)
Did you mean?  gitlab:packages:migrate
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
(See full trace by running task with --trace)

After some time (overnight? the weekend?) the background jobs completed. How do I make these migration tasks execute immediately and not in the background? This is important!

Hi,

Sometimes the migrations can take longer, this is documented in the upgrade guide. Personally, I’ve never had issues with it, and they have always finished very quickly after upgrades. I’ve never had a situation where they have taken days.

As a rule of thumb, any database smaller than 10 GB doesn’t take too much time to upgrade; perhaps an hour at most per minor release. Larger databases however may require more time, but this is highly dependent on the size of the database and the migrations that are being performed.

Hi,

these jobs run intentionally in the background, as they might block the application when run immediately.

See this section in the 14.0 release post. There was a problem in 14.0 with these batched jobs not starting, this was addressed in 14.0.3, so you are not affected by this. You’ll “only” need to wait until the database migrations are finished. I have no better answer, sorry

Cheers,
Michael

@dnsmichi Thanks. I must do the whole thing off-line because I need to patch gitlab with a feature or two. Is there no way to ‘foreground’ these jobs? They take quite a bit longer than @iwalker 's suggestion