We’re upgrading from 12.7 to 14.2 via Omnibus package installation (CentOS x86_64). I upgraded to 12.10.14, 13.0.14, 13.1.11, 13.12.8, and now 14.0.6. At each step, gitlab reconfigure was run, and all background jobs/migrations had completed before proceding. We are stuck now.
Background migrations page https://gitlab-test.local/admin/background_migrations shows 11 migration tasks all at 0.00% and status Active. These include items such as:
- CopyColumnUsingBackgroundMigrationJob: ci_build_trace_sections
- CopyColumnUsingBackgroundMigrationJob: ci_builds_metadata
There are 4 that have finished. There are no jobs in the “busy” queue under monitoring/background jobs.
The Troubleshooting section of the omnibus installer does not help here.
I tried the 14.1 docs to no avail (See below).
What can I do to facilitate the execution of these jobs?
Gitlab::Database::BackgroundMigration::BatchedMigration.pluck(:id, :table_name, :status)
=> [[1, "ci_build_trace_sections", "active"], [2, "ci_builds_metadata", "active"], [3, "events", "active"], [4, "push_event_payloads", "active"], [5, "ci_job_artifacts", "active"], [6, "ci_sources_pipelines", "finished"], [7, "ci_build_needs", "active"], [8, "ci_builds", "active"], [9, "ci_builds_runner_session", "finished"], [10, "ci_build_trace_chunks", "finished"], [11, "taggings", "active"], [12, "deployments", "active"], [13, "geo_job_artifact_deleted_events", "finished"], [14, "ci_stages", "active"], [15, "ci_builds_metadata", "active"]]
sushi:[gitlab-test]/etc/gitlab # gitlab-rake gitlab:background_migrations:finalize[CopyColumnUsingBackgroundMigrationJob,push_event_payloads,event_id,'[["event_id"]\, ["event_id_convert_to_bigint"]]']
rake aborted!
Don't know how to build task 'gitlab:background_migrations:finalize' (See the list of available tasks with `rake --tasks`)
Did you mean? gitlab:packages:migrate
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
(See full trace by running task with --trace)