Here’s a summary of where this issue currently stands:
I’m working on upgrading a gitlab omnibus installation (v12.3.0-ce) on a VM instance. Our goal is to ultimately upgrade to the latest version of gitlab (v13.x). According to the documentation we need to first upgrade to gitlab version 12.10.14 and there must be no active background migrations.
Once the server is upgraded, we execute the following command to reconfigure the system sudo gitlab-ctl reconfigure . This process is fine until gitlab version 12.9.0-ce.0 at which point Sidekiq perpetually requeues the same background migration that produce the following errors in its logs:
{"severity":"WARN","time":"2021-06-04T20:56:27.849Z","class":"BackgroundMigrationWorker","retry":3,"queue":"background_migration","jid":"287afd8f9d56274f68ab5b99","created_at":"2021-06-04T20:54:25.804Z","meta.caller_id":"UserMentions::CreateResourceUserMention","correlation_id":"f6039eaf71ec8985f153c79e16e784dc","enqueued_at":"2021-06-04T20:56:27.680Z","pid":12076,"message":"BackgroundMigrationWorker JID-287afd8f9d56274f68ab5b99: fail: 0.16901 sec","job_status":"fail","scheduling_latency_s":0.000927,"duration":0.16901,"cpu_s":0.131661,"completed_at":"2021-06-04T20:56:27.849Z","error_message":"Sidekiq::JobRetry::Skip","error_class":"Sidekiq::JobRetry::Skip","db_duration":42.51408199525031,"db_duration_s":0.04251408199525031}
{"severity":"WARN","time":"2021-06-04T20:56:27.851Z","error_class":"ArgumentError","error_message":"must specify an iv","context":"Job raised exception","jobstr":"{\"class\":\"BackgroundMigrationWorker\",\"args\":[\"UserMentions::CreateResourceUserMention\",[\"MergeRequest\",\"LEFT JOIN merge_request_user_mentions on merge_requests.id = merge_request_user_mentions.merge_request_id\",\"(description like '%@%' OR title like '%@%') AND merge_request_user_mentions.merge_request_id IS NULL\",false,384,4034]],\"retry\":3,\"queue\":\"background_migration\",\"jid\":\"287afd8f9d56274f68ab5b99\",\"created_at\":1622840065.804012,\"meta.caller_id\":\"UserMentions::CreateResourceUserMention\",\"correlation_id\":\"f6039eaf71ec8985f153c79e16e784dc\",\"enqueued_at\":1622840187.6800113}","class":"BackgroundMigrationWorker","args":["UserMentions::CreateResourceUserMention",["MergeRequest","LEFT JOIN merge_request_user_mentions on merge_requests.id = merge_request_user_mentions.merge_request_id","(description like '%@%' OR title like '%@%') AND merge_request_user_mentions.merge_request_id IS NULL",false,384,4034]],"retry":3,"queue":"background_migration","jid":"287afd8f9d56274f68ab5b99","created_at":"2021-06-04T20:54:25.804Z","meta.caller_id":"UserMentions::CreateResourceUserMention","correlation_id":"f6039eaf71ec8985f153c79e16e784dc","enqueued_at":"2021-06-04T20:56:27.680Z","error_backtrace":["app/models/project_services/data_fields.rb:14:in `url'","app/models/project_services/jira_service.rb:131:in `url'","app/models/project_services/jira_service.rb:122:in `issues_url'","app/models/project_services/issue_tracker_service.rb:88:in `issue_url'","app/models/project_services/issue_tracker_service.rb:100:in `issue_path'","app/helpers/issues_helper.rb:37:in `url_for_tracker_issue'","app/helpers/issues_helper.rb:26:in `url_for_issue'","lib/banzai/filter/external_issue_reference_filter.rb:86:in `url_for_issue'","lib/banzai/filter/external_issue_reference_filter.rb:71:in `block in issue_link_filter'","lib/banzai/filter/external_issue_reference_filter.rb:26:in `block in references_in'","lib/banzai/filter/external_issue_reference_filter.rb:25:in `gsub'","lib/banzai/filter/external_issue_reference_filter.rb:25:in `references_in'","lib/banzai/filter/external_issue_reference_filter.rb:68:in `issue_link_filter'","lib/banzai/filter/external_issue_reference_filter.rb:40:in `block (2 levels) in call'","lib/banzai/filter/reference_filter.rb:121:in `replace_text_when_pattern_matches'","lib/banzai/filter/external_issue_reference_filter.rb:39:in `block in call'","lib/banzai/filter/reference_filter.rb:98:in `block in each_node'","lib/banzai/filter/reference_filter.rb:97:in `each_node'","lib/banzai/filter/external_issue_reference_filter.rb:37:in `call'","lib/banzai/pipeline/base_pipeline.rb:23:in `block (2 levels) in singleton class'","lib/banzai/renderer.rb:130:in `render_result'","lib/banzai/renderer.rb:161:in `block in cacheless_render'","lib/banzai/renderer.rb:160:in `cacheless_render'","lib/banzai/renderer.rb:52:in `cacheless_render_field'","app/models/concerns/cache_markdown_field.rb:49:in `rendered_field_content'","app/models/concerns/cache_markdown_field.rb:58:in `block in refresh_markdown_cache'","app/models/concerns/cache_markdown_field.rb:55:in `map'","app/models/concerns/cache_markdown_field.rb:55:in `refresh_markdown_cache'","app/models/concerns/cache_markdown_field.rb:105:in `updated_cached_html_for'","lib/gitlab/background_migration/user_mentions/models/concerns/isolated_mentionable.rb:52:in `block in all_references'","lib/gitlab/background_migration/user_mentions/models/concerns/isolated_mentionable.rb:44:in `each'","lib/gitlab/background_migration/user_mentions/models/concerns/isolated_mentionable.rb:44:in `all_references'","lib/gitlab/background_migration/user_mentions/models/concerns/isolated_mentionable.rb:70:in `build_mention_values'","lib/gitlab/background_migration/user_mentions/create_resource_user_mention.rb:24:in `block (2 levels) in perform'","lib/gitlab/background_migration/user_mentions/create_resource_user_mention.rb:23:in `each'","lib/gitlab/background_migration/user_mentions/create_resource_user_mention.rb:23:in `block in perform'","lib/gitlab/background_migration/user_mentions/create_resource_user_mention.rb:21:in `each'","lib/gitlab/background_migration/user_mentions/create_resource_user_mention.rb:21:in `perform'","lib/gitlab/background_migration.rb:58:in `perform'","app/workers/background_migration_worker.rb:29:in `block in perform'","app/workers/concerns/worker_context.rb:63:in `block in with_context'","lib/gitlab/application_context.rb:46:in `block in use'","lib/gitlab/application_context.rb:46:in `use'","app/workers/concerns/worker_context.rb:63:in `with_context'","app/workers/background_migration_worker.rb:25:in `perform'","lib/gitlab/with_request_store.rb:7:in `with_request_store'","lib/gitlab/sidekiq_daemon/monitor.rb:49:in `within_job'"]}
This blocks further upgrades, (all background migrations must have executed successfully). Does anyone have any suggestion on what can be done to get the migrations to execute successfully? Any suggestions on how to further troubleshoot this issue would also be appreciated.