I’m lost, the upgrade failed and I can’t get it running. Please help.
Attempting to run '/scripts/wait-for-deps /scripts/db-migrate' as a main process
Checking database migrations are up-to-date
Performing migrations (this will initialized if needed)
rake aborted!
StandardError: An error has occurred, this and all later migrations canceled:
Expected batched background migration for the given configuration to be marked as 'finished', but it is 'failed': { :job_class_name=>"ProjectNamespaces::BackfillProjectNamespaces", :table_name=>:projects, :column_name=>:id, :job_arguments=>[nil, "up"] }
Finalize it manualy by running
sudo gitlab-rake gitlab:background_migrations:finalize[ProjectNamespaces::BackfillProjectNamespaces, projects, id, '[null,"up"]']
For more information, check the documentation
https://docs.gitlab.com/ee/user/admin_area/monitoring/background_migrations.html#database-migrations-failing-because-of-batched-background-migration-not-finished
I’ve tried running the command shown in the logs using the toolbox pod with no success.
gitlab-rake gitlab:background_migrations:finalize[ProjectNamespaces::BackfillProjectNamespaces, projects, id, '[null,"up"]']
or
gitlab-rake 'gitlab:background_migrations:finalize[ProjectNamespaces::BackfillProjectNamespaces, projects, id, [null\, "up"]]'
Error:
git@gitlab-toolbox-7744fc7ddd-8444b:/$ gitlab-rake 'gitlab:background_migrations:finalize[ProjectNamespaces::BackfillProjectNamespaces, projects, id, [null\, "up"]]'
rake aborted!
Gitlab::Database::BackgroundMigration::BatchedMigrationRunner::FailedToFinalize: Gitlab::Database::BackgroundMigration::BatchedMigrationRunner::FailedToFinalize
/srv/gitlab/lib/gitlab/database/background_migration/batched_migration_runner.rb:76:in `finalize'
/srv/gitlab/lib/gitlab/database/background_migration/batched_migration_runner.rb:10:in `finalize'
/srv/gitlab/lib/tasks/gitlab/background_migrations.rake:70:in `finalize_migration'
/srv/gitlab/lib/tasks/gitlab/background_migrations.rake:18:in `block (3 levels) in <main>'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/sentry-ruby-core-5.1.1/lib/sentry/rake.rb:26:in `execute'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/rake-13.0.6/exe/rake:27:in `<top (required)>'
/srv/gitlab/bin/bundle:5:in `load'
/srv/gitlab/bin/bundle:5:in `<main>'
Tasks: TOP => gitlab:background_migrations:finalize
(See full trace by running task with --trace)
Checked background jobs and there are none
gitlab-rails runner -e production 'puts Gitlab::Database::BackgroundMigrationJob.pending'
Running gitlab-rake gitlab:background_migrations:status
git@gitlab-toolbox-7744fc7ddd-8444b:/$ gitlab-rake gitlab:background_migrations:status
Database: main
finished | CopyColumnUsingBackgroundMigrationJob,ci_builds_metadata,id,[["build_id"],["build_id_convert_to_bigint"]]
finished | CopyColumnUsingBackgroundMigrationJob,events,id,[["id"],["id_convert_to_bigint"]]
finished | CopyColumnUsingBackgroundMigrationJob,push_event_payloads,event_id,[["event_id"],["event_id_convert_to_bigint"]]
finished | CopyColumnUsingBackgroundMigrationJob,ci_job_artifacts,id,[["id","job_id"],["id_convert_to_bigint","job_id_convert_to_bigint"]]
finished | CopyColumnUsingBackgroundMigrationJob,ci_sources_pipelines,id,[["source_job_id"],["source_job_id_convert_to_bigint"]]
finished | CopyColumnUsingBackgroundMigrationJob,ci_build_needs,id,[["build_id"],["build_id_convert_to_bigint"]]
finished | CopyColumnUsingBackgroundMigrationJob,ci_builds,id,[["id","stage_id"],["id_convert_to_bigint","stage_id_convert_to_bigint"]]
finished | CopyColumnUsingBackgroundMigrationJob,ci_builds_runner_session,id,[["build_id"],["build_id_convert_to_bigint"]]
finished | CopyColumnUsingBackgroundMigrationJob,ci_build_trace_chunks,id,[["build_id"],["build_id_convert_to_bigint"]]
finished | CopyColumnUsingBackgroundMigrationJob,taggings,id,[["id","taggable_id"],["id_convert_to_bigint","taggable_id_convert_to_bigint"]]
finished | CopyColumnUsingBackgroundMigrationJob,deployments,id,[["deployable_id"],["deployable_id_convert_to_bigint"]]
finished | CopyColumnUsingBackgroundMigrationJob,geo_job_artifact_deleted_events,id,[["job_artifact_id"],["job_artifact_id_convert_to_bigint"]]
finished | CopyColumnUsingBackgroundMigrationJob,ci_stages,id,[["id"],["id_convert_to_bigint"]]
finished | CopyColumnUsingBackgroundMigrationJob,ci_builds_metadata,id,[["id"],["id_convert_to_bigint"]]
finished | BackfillUserNamespace,namespaces,id,[]
failed | BackfillIssueSearchData,issues,id,[]
failed | BackfillNamespaceIdForNamespaceRoute,routes,id,[]
failed | BackfillMemberNamespaceForGroupMembers,members,id,[]
failed | MigratePersonalNamespaceProjectMaintainerToOwner,members,id,[]
failed | NullifyOrphanRunnerIdOnCiBuilds,ci_builds,id,[]
failed | ProjectNamespaces::BackfillProjectNamespaces,projects,id,[null,"up"]
failed | BackfillGroupFeatures,namespaces,id,[10000]
active | BackfillWorkItemTypeIdForIssues,issues,id,[0,1]
active | BackfillWorkItemTypeIdForIssues,issues,id,[1,2]
active | BackfillWorkItemTypeIdForIssues,issues,id,[2,3]
active | BackfillWorkItemTypeIdForIssues,issues,id,[3,4]
active | BackfillWorkItemTypeIdForIssues,issues,id,[4,5]
active | BackfillIntegrationsTypeNew,integrations,id,[]
