Hi,
after upgrading gitlab-ce from 15.3.2 to 15.11.0 via RPM on RHEL8, I have two failing background migrations.
CopyColumnUsingBackgroundMigrationJob: ci_build_needs
CopyColumnUsingBackgroundMigrationJob: sent_notifications
Both are more or less failing with a similar error:
ERROR: column \"id_convert_to_bigint\" of relation \"ci_build_needs\" does not exist\nLINE 1: ...db_config_name:main*/ UPDATE \"ci_build_needs\" SET \"id_conver...\n ^\n","exception.backtrace":["lib/gitlab/database/load_balancing/connection_proxy.rb:121:in `public_send'","lib/gitlab/database/load_balancing/connection_proxy.rb:121:in `block in write_using_load_balancer'","lib/gitlab/database/load_balancing/load_balancer.rb:127:in `block in read_write'","lib/gitlab/database/load_balancing/load_balancer.rb:198:in `retry_with_backoff'","lib/gitlab/database/load_balancing/load_balancer.rb:116:in `read_write'","lib/gitlab/database/load_balancing/connection_proxy.rb:120:in `write_using_load_balancer'","lib/gitlab/database/load_balancing/connection_proxy.rb:62:in `block (2 levels) in \u003cclass:ConnectionProxy\u003e'","lib/gitlab/background_migration/copy_column_using_background_migration_job.rb:25:in `block in perform'","lib/gitlab/background_migration/batched_migration_job.rb:105:in `block (2 levels) in each_sub_batch'","lib/gitlab/database/background_migration/batch_metrics.rb:22:in `instrument_operation'","lib/gitlab/background_migration/batched_migration_job.rb:104:in `block in each_sub_batch'","app/models/concerns/each_batch.rb:99:in `block (2 levels) in each_batch'","app/models/concerns/each_batch.rb:99:in `block in each_batch'","app/models/concerns/each_batch.rb:69:in `step'","app/models/concerns/each_batch.rb:69:in `each_batch'","lib/gitlab/background_migration/batched_migration_job.rb:103:in `each_sub_batch'","lib/gitlab/background_migration/copy_column_using_background_migration_job.rb:24:in `perform'","lib/gitlab/database/background_migration/batched_migration_wrapper.rb:81:in `execute_batched_migration_job'","lib/gitlab/database/background_migration/batched_migration_wrapper.rb:63:in `execute_job'","lib/gitlab/database/background_migration/batched_migration_wrapper.rb:50:in `execute_batch'","lib/gitlab/database/background_migration/batched_migration_wrapper.rb:25:in `perform'","lib/gitlab/database/background_migration/batched_migration_runner.rb:30:in `run_migration_job'","app/workers/database/batched_background_migration/execution_worker.rb:103:in `run_migration_job'","app/workers/database/batched_background_migration/execution_worker.rb:56:in `block (2 levels) in perform_work'","app/services/concerns/exclusive_lease_guard.rb:29:in `try_obtain_lease'","app/workers/database/batched_background_migration/execution_worker.rb:55:in `block in perform_work'","lib/gitlab/database/shared_model.rb:34:in `using_connection'","app/workers/database/batched_background_migration/execution_worker.rb:50:in `perform_work'","app/workers/concerns/limited_capacity/worker.rb:94:in `perform_registered'","app/workers/concerns/limited_capacity/worker.rb:65:in `perform'","lib/gitlab/database/load_balancing/sidekiq_server_middleware.rb:26:in `call'","lib/gitlab/sidekiq_middleware/duplicate_jobs/strategies/none.rb:14:in `perform'","lib/gitlab/sidekiq_middleware/duplicate_jobs/duplicate_job.rb:44:in `perform'","lib/gitlab/sidekiq_middleware/duplicate_jobs/server.rb:8:in `call'","lib/gitlab/sidekiq_middleware/worker_context.rb:9:in `wrap_in_optional_context'","lib/gitlab/sidekiq_middleware/worker_context/server.rb:19:in `block in call'","lib/gitlab/application_context.rb:118:in `block in use'","lib/gitlab/application_context.rb:118:in `use'","lib/gitlab/application_context.rb:57:in `with_context'","lib/gitlab/sidekiq_middleware/worker_context/server.rb:17:in `call'","lib/gitlab/sidekiq_status/server_middleware.rb:7:in `call'","lib/gitlab/sidekiq_versioning/middleware.rb:9:in `call'","lib/gitlab/sidekiq_middleware/query_analyzer.rb:7:in `block in call'","lib/gitlab/database/query_analyzer.rb:37:in `within'","lib/gitlab/sidekiq_middleware/query_analyzer.rb:7:in `call'","lib/gitlab/sidekiq_middleware/admin_mode/server.rb:14:in `call'","lib/gitlab/sidekiq_middleware/instrumentation_logger.rb:9:in `call'","lib/gitlab/sidekiq_middleware/batch_loader.rb:7:in `call'","lib/gitlab/sidekiq_middleware/extra_done_log_metadata.rb:7:in `call'","lib/gitlab/sidekiq_middleware/request_store_middleware.rb:10:in `block in call'","lib/gitlab/with_request_store.rb:17:in `enabling_request_store'","lib/gitlab/with_request_store.rb:10:in `with_request_store'","lib/gitlab/sidekiq_middleware/request_store_middleware.rb:9:in `call'","lib/gitlab/sidekiq_middleware/server_metrics.rb:76:in `block in call'","lib/gitlab/sidekiq_middleware/server_metrics.rb:103:in `block in instrument'","lib/gitlab/metrics/background_transaction.rb:33:in `run'","lib/gitlab/sidekiq_middleware/server_metrics.rb:103:in `instrument'","lib/gitlab/sidekiq_middleware/server_metrics.rb:75:in `call'","lib/gitlab/sidekiq_middleware/monitor.rb:10:in `block in call'","lib/gitlab/sidekiq_daemon/monitor.rb:46:in `within_job'","lib/gitlab/sidekiq_middleware/monitor.rb:9:in `call'","lib/gitlab/sidekiq_middleware/size_limiter/server.rb:13:in `call'","lib/gitlab/sidekiq_logging/structured_logger.rb:21:in `call'"],"exception.cause_class":"PG::UndefinedColumn","exception.sql":"/*application:sidekiq,correlation_id:fa6c494b55c29dbdd841b55804c0f315,jid:0019fe071b838484e1642637,endpoint_id:Database::BatchedBackgroundMigration::MainExecutionWorker,db_config_name:main*/ UPDATE \"ci_build_needs\" SET \"id_convert_to_bigint\" = \"id\" WHERE \"ci_build_needs\".\"id\" BETWEEN $1 AND $2 AND \"ci_build_needs\".\"id\" \u003e= $3","db_duration_s":0.036358}
I am not sure why the column id_convert_to_bigint never got created.
How would I go about fixing this though?
Thanks
Kristian