Hi!
After upgrading from 13.12.12 to 14.0.11 , 7 background migrations were stuck at 0% progress.
job_class_name | table_name
---------------------------------------+---------------------
CopyColumnUsingBackgroundMigrationJob | ci_builds_metadata
CopyColumnUsingBackgroundMigrationJob | ci_stages
CopyColumnUsingBackgroundMigrationJob | ci_builds_metadata
CopyColumnUsingBackgroundMigrationJob | push_event_payloads
CopyColumnUsingBackgroundMigrationJob | events
CopyColumnUsingBackgroundMigrationJob | ci_job_artifacts
CopyColumnUsingBackgroundMigrationJob | ci_builds
Manually finalizing appeared to work for 6 of them exept ci_builds_metadata, but the migrations status seem incorrect in the database although those 6 don’t appear on the WebUI anymore (not even in Failed or Finished lists).
gitlabhq_production=# select job_class_name, status, table_name, column_name, job_arguments from batched_background_migrations where status <> 3;
job_class_name | status | table_name | column_name | job_arguments
---------------------------------------+--------+---------------------+-------------+------------------------------------------------------------------------------
CopyColumnUsingBackgroundMigrationJob | 1 | ci_builds_metadata | id | [["id"], ["id_convert_to_bigint"]]
CopyColumnUsingBackgroundMigrationJob | 5 | ci_stages | id | [["id"], ["id_convert_to_bigint"]]
CopyColumnUsingBackgroundMigrationJob | 5 | ci_builds_metadata | id | [["build_id"], ["build_id_convert_to_bigint"]]
CopyColumnUsingBackgroundMigrationJob | 5 | push_event_payloads | event_id | [["event_id"], ["event_id_convert_to_bigint"]]
CopyColumnUsingBackgroundMigrationJob | 5 | events | id | [["id"], ["id_convert_to_bigint"]]
CopyColumnUsingBackgroundMigrationJob | 5 | ci_job_artifacts | id | [["id", "job_id"], ["id_convert_to_bigint", "job_id_convert_to_bigint"]]
CopyColumnUsingBackgroundMigrationJob | 5 | ci_builds | id | [["id", "stage_id"], ["id_convert_to_bigint", "stage_id_convert_to_bigint"]]
Trying to finalize ci_builds_metadata throws the following errors (stack trace and some repeated errors truncated):
rake aborted!
ActiveModel::MissingAttributeError: can't write unknown attribute ``
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activemodel-6.1.3.2/lib/active_model/attribute.rb:207:in `with_value_from_database'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activemodel-6.1.3.2/lib/active_model/attribute_set.rb:51:in `write_from_user'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/transactions.rb:415:in `restore_transaction_record_state'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/transactions.rb:334:in `rolledback!'
[...]
Caused by:
ActiveRecord::StatementInvalid: PG::SyntaxError: ERROR: zero-length delimited identifier at or near """"
LINE 1: ...38.662761' WHERE "batched_background_migration_jobs"."" = 24
[...]
Caused by:
PG::SyntaxError: ERROR: zero-length delimited identifier at or near """"
LINE 1: ...38.662761' WHERE "batched_background_migration_jobs"."" = 24
[...]
Caused by:
ActiveModel::MissingAttributeError: can't write unknown attribute ``
[...]
Caused by:
ActiveRecord::StatementInvalid: PG::SyntaxError: ERROR: zero-length delimited identifier at or near """"
LINE 1: ...empts" = 1 WHERE "batched_background_migration_jobs"."" = 24
[...]
Migration process allowed me to upgrade to 14.1.0 but next minor version needs those migrations to be properly finished.
I thought of dropping and recreating those tables as they don’t contain any vital information but the different relationships with other data seem to render that method impossible.
I have a test copy and free to break it as much as needed for testing.
Open to any suggestion!