Upgrade to 14.0.11: Background migrations stuck, manual finalize failed

Hi!

After upgrading from 13.12.12 to 14.0.11 , 7 background migrations were stuck at 0% progress.

            job_class_name             |     table_name      
---------------------------------------+---------------------
 CopyColumnUsingBackgroundMigrationJob | ci_builds_metadata
 CopyColumnUsingBackgroundMigrationJob | ci_stages
 CopyColumnUsingBackgroundMigrationJob | ci_builds_metadata
 CopyColumnUsingBackgroundMigrationJob | push_event_payloads
 CopyColumnUsingBackgroundMigrationJob | events
 CopyColumnUsingBackgroundMigrationJob | ci_job_artifacts
 CopyColumnUsingBackgroundMigrationJob | ci_builds

Manually finalizing appeared to work for 6 of them exept ci_builds_metadata, but the migrations status seem incorrect in the database although those 6 don’t appear on the WebUI anymore (not even in Failed or Finished lists).

gitlabhq_production=# select job_class_name, status, table_name, column_name, job_arguments from batched_background_migrations where status <> 3;
            job_class_name             | status |     table_name      | column_name |                                job_arguments                                 
---------------------------------------+--------+---------------------+-------------+------------------------------------------------------------------------------
 CopyColumnUsingBackgroundMigrationJob |      1 | ci_builds_metadata  | id          | [["id"], ["id_convert_to_bigint"]]
 CopyColumnUsingBackgroundMigrationJob |      5 | ci_stages           | id          | [["id"], ["id_convert_to_bigint"]]
 CopyColumnUsingBackgroundMigrationJob |      5 | ci_builds_metadata  | id          | [["build_id"], ["build_id_convert_to_bigint"]]
 CopyColumnUsingBackgroundMigrationJob |      5 | push_event_payloads | event_id    | [["event_id"], ["event_id_convert_to_bigint"]]
 CopyColumnUsingBackgroundMigrationJob |      5 | events              | id          | [["id"], ["id_convert_to_bigint"]]
 CopyColumnUsingBackgroundMigrationJob |      5 | ci_job_artifacts    | id          | [["id", "job_id"], ["id_convert_to_bigint", "job_id_convert_to_bigint"]]
 CopyColumnUsingBackgroundMigrationJob |      5 | ci_builds           | id          | [["id", "stage_id"], ["id_convert_to_bigint", "stage_id_convert_to_bigint"]]

Trying to finalize ci_builds_metadata throws the following errors (stack trace and some repeated errors truncated):

rake aborted!
ActiveModel::MissingAttributeError: can't write unknown attribute ``
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activemodel-6.1.3.2/lib/active_model/attribute.rb:207:in `with_value_from_database'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activemodel-6.1.3.2/lib/active_model/attribute_set.rb:51:in `write_from_user'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/transactions.rb:415:in `restore_transaction_record_state'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/transactions.rb:334:in `rolledback!'
[...]

Caused by:
ActiveRecord::StatementInvalid: PG::SyntaxError: ERROR:  zero-length delimited identifier at or near """"
LINE 1: ...38.662761' WHERE "batched_background_migration_jobs"."" = 24

[...]  

Caused by:
PG::SyntaxError: ERROR:  zero-length delimited identifier at or near """"
LINE 1: ...38.662761' WHERE "batched_background_migration_jobs"."" = 24

[...]

Caused by:
ActiveModel::MissingAttributeError: can't write unknown attribute ``
[...]

Caused by:
ActiveRecord::StatementInvalid: PG::SyntaxError: ERROR:  zero-length delimited identifier at or near """"
LINE 1: ...empts" = 1 WHERE "batched_background_migration_jobs"."" = 24
[...]

Migration process allowed me to upgrade to 14.1.0 but next minor version needs those migrations to be properly finished.

I thought of dropping and recreating those tables as they don’t contain any vital information but the different relationships with other data seem to render that method impossible.

I have a test copy and free to break it as much as needed for testing.

Open to any suggestion!