Hello,
I have an ubuntu server 18.04 LTS with Gitlab installed. I wanted to upgrade the gitlab version (I think I had 13.X or so) to the newest one. The installation aborted because I should first install 14.0. I did this (no errors) and then I wanted to install straight to 14.3.2 (by using apt-get dist-upgrade). Unfortunately some migrations scripts failed:

Running handlers:
There was an error running gitlab-ctl reconfigure:

rails_migration[gitlab-rails] (gitlab::database_migrations line 51) had an error: Mixlib::ShellOut::ShellCommandFailed: bash[migrate gitlab-rails database] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/resources/rails_migration.rb line 16) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
---- Begin output of "bash"  "/tmp/chef-script20211005-2430-18bo7hn" ----
STDOUT: == 20210701141346 FinalizeCiBuildsStageIdBigintConversion: migrating ==========
-- transaction_open?()
   -> 0.0000s
-- index_exists?("ci_builds", :stage_id_convert_to_bigint, {:name=>:index_ci_builds_on_converted_stage_id, :algorithm=>:concurrently})
   -> 0.0152s
-- execute("SET statement_timeout TO 0")
   -> 0.0003s
-- add_index("ci_builds", :stage_id_convert_to_bigint, {:name=>:index_ci_builds_on_converted_stage_id, :algorithm=>:concurrently})
   -> 0.0118s
-- execute("RESET statement_timeout")
   -> 0.0003s
-- transaction_open?()
   -> 0.0000s
-- foreign_keys("ci_builds")
   -> 0.0058s
-- execute("LOCK TABLE ci_stages, ci_builds IN SHARE ROW EXCLUSIVE MODE")
   -> 0.0004s
-- execute("ALTER TABLE ci_builds\nADD CONSTRAINT fk_c050b39299\nFOREIGN KEY (stage_id_convert_to_bigint)\nREFERENCES ci_stages (id)\nON DELETE CASCADE\nNOT VALID;\n")
   -> 0.0020s
-- execute("SET statement_timeout TO 0")
   -> 0.0011s
-- execute("ALTER TABLE ci_builds VALIDATE CONSTRAINT fk_c050b39299;")
   -> 0.0066s
-- execute("RESET statement_timeout")
   -> 0.0003s
-- quote_table_name("ci_builds")
   -> 0.0000s
-- quote_table_name(:ci_stages)
   -> 0.0000s
-- execute("LOCK TABLE \"ci_stages\", \"ci_builds\" IN ACCESS EXCLUSIVE MODE")
   -> 0.0003s
-- quote_column_name(:stage_id)
   -> 0.0000s
-- quote_column_name("stage_id_tmp")
   -> 0.0000s
-- execute("ALTER TABLE \"ci_builds\" RENAME COLUMN \"stage_id\" TO \"stage_id_tmp\"")
   -> 0.0004s
-- quote_column_name(:stage_id_convert_to_bigint)
   -> 0.0000s
-- quote_column_name(:stage_id)
   -> 0.0000s
-- execute("ALTER TABLE \"ci_builds\" RENAME COLUMN \"stage_id_convert_to_bigint\" TO \"stage_id\"")
   -> 0.0011s
-- quote_column_name("stage_id_tmp")
   -> 0.0000s
-- quote_column_name(:stage_id_convert_to_bigint)
   -> 0.0000s
-- execute("ALTER TABLE \"ci_builds\" RENAME COLUMN \"stage_id_tmp\" TO \"stage_id_convert_to_bigint\"")
   -> 0.0004s
-- quote_table_name("trigger_3f6129be01d2")
   -> 0.0000s
-- execute("ALTER FUNCTION \"trigger_3f6129be01d2\" RESET ALL")
   -> 0.0004s
-- execute("DROP INDEX index_ci_builds_on_stage_id")
   -> 0.0006s
-- rename_index("ci_builds", :index_ci_builds_on_converted_stage_id, :index_ci_builds_on_stage_id)
   -> 0.0004s
-- remove_foreign_key("ci_builds", {:name=>"fk_3a9eaa254d"})
   -> 0.0072s
-- quote_table_name("ci_builds")
   -> 0.0000s
-- quote_column_name("fk_c050b39299")
   -> 0.0000s
-- quote_column_name("fk_3a9eaa254d")
   -> 0.0000s
-- execute("ALTER TABLE \"ci_builds\"\nRENAME CONSTRAINT \"fk_c050b39299\" TO \"fk_3a9eaa254d\"\n")
   -> 0.0003s
== 20210701141346 FinalizeCiBuildsStageIdBigintConversion: migrated (0.1338s) =

== 20210702124842 AddCiJobTraceSizeToPlanLimits: migrating ====================
-- add_column(:plan_limits, :ci_jobs_trace_size_limit, :integer, {:default=>100, :null=>false})
   -> 0.0025s
== 20210702124842 AddCiJobTraceSizeToPlanLimits: migrated (0.0025s) ===========

== 20210705124128 AddProjectSettingsPreviousDefaultBranch: migrating ==========
-- add_column(:project_settings, :previous_default_branch, :text)
   -> 0.0079s
== 20210705124128 AddProjectSettingsPreviousDefaultBranch: migrated (0.0105s) =

== 20210705130919 CreateContainerReposOnExpCleanupStatusProjectIdStartDateIndex: migrating 
-- transaction_open?()
   -> 0.0000s
-- index_exists?(:container_repositories, [:expiration_policy_cleanup_status, :project_id, :expiration_policy_started_at], {:name=>"idx_container_repos_on_exp_cleanup_status_project_id_start_date", :algorithm=>:concurrently})
   -> 0.0039s
-- execute("SET statement_timeout TO 0")
   -> 0.0002s
-- add_index(:container_repositories, [:expiration_policy_cleanup_status, :project_id, :expiration_policy_started_at], {:name=>"idx_container_repos_on_exp_cleanup_status_project_id_start_date", :algorithm=>:concurrently})
   -> 0.0052s
-- execute("RESET statement_timeout")
   -> 0.0002s
-- transaction_open?()
   -> 0.0000s
-- index_exists?(:container_repositories, [:expiration_policy_cleanup_status, :expiration_policy_started_at], {:name=>"idx_container_repositories_on_exp_cleanup_status_and_start_date", :algorithm=>:concurrently})
   -> 0.0040s
-- execute("SET statement_timeout TO 0")
   -> 0.0002s
-- remove_index(:container_repositories, {:name=>"idx_container_repositories_on_exp_cleanup_status_and_start_date", :algorithm=>:concurrently, :column=>[:expiration_policy_cleanup_status, :expiration_policy_started_at]})
   -> 0.0068s
-- execute("RESET statement_timeout")
   -> 0.0003s
== 20210705130919 CreateContainerReposOnExpCleanupStatusProjectIdStartDateIndex: migrated (0.0232s) 

== 20210705132928 AddNewUserSignupsCapToNamespaceSettings: migrating ==========
-- add_column(:namespace_settings, :new_user_signups_cap, :integer, {:null=>true})
   -> 0.0008s
== 20210705132928 AddNewUserSignupsCapToNamespaceSettings: migrated (0.0034s) =

== 20210705144657 AddInstanceRunnersEnabledToCiPendingBuild: migrating ========
-- add_column(:ci_pending_builds, :instance_runners_enabled, :boolean, {:null=>false, :default=>false})
   -> 0.0011s
== 20210705144657 AddInstanceRunnersEnabledToCiPendingBuild: migrated (0.0050s) 

== 20210706084713 AddDevopsAdoptionSnapshotsIndex: migrating ==================
-- transaction_open?()
   -> 0.0000s
-- index_exists?(:analytics_devops_adoption_snapshots, [:namespace_id, :end_time], {:where=>"recorded_at >= end_time", :name=>"idx_analytics_devops_adoption_snapshots_finalized", :algorithm=>:concurrently})
   -> 0.0036s
-- execute("SET statement_timeout TO 0")
   -> 0.0003s
-- add_index(:analytics_devops_adoption_snapshots, [:namespace_id, :end_time], {:where=>"recorded_at >= end_time", :name=>"idx_analytics_devops_adoption_snapshots_finalized", :algorithm=>:concurrently})
   -> 0.0052s
-- execute("RESET statement_timeout")
   -> 0.0003s
== 20210706084713 AddDevopsAdoptionSnapshotsIndex: migrated (0.0104s) =========

== 20210706112800 RemoveCloudLicenseEnabledFromApplicationSettings: migrating =
-- remove_column(:application_settings, :cloud_license_enabled)
   -> 0.0034s
== 20210706112800 RemoveCloudLicenseEnabledFromApplicationSettings: migrated (0.0078s) 

== 20210706115312 AddUpvotesCountIndexToIssues: migrating =====================
-- transaction_open?()
   -> 0.0000s
-- index_exists?(:issues, [:project_id, :upvotes_count], {:name=>"index_issues_on_project_id_and_upvotes_count", :algorithm=>:concurrently})
   -> 0.0136s
-- execute("SET statement_timeout TO 0")
   -> 0.0002s
-- add_index(:issues, [:project_id, :upvotes_count], {:name=>"index_issues_on_project_id_and_upvotes_count", :algorithm=>:concurrently})
   -> 0.0049s
-- execute("RESET statement_timeout")
   -> 0.0002s
== 20210706115312 AddUpvotesCountIndexToIssues: migrated (0.0209s) ============

== 20210706120847 RemoveFrameworkColumnFromComplianceManagementFrameworks: migrating 
-- remove_column(:project_compliance_framework_settings, :framework, :smallint)
   -> 0.0011s
== 20210706120847 RemoveFrameworkColumnFromComplianceManagementFrameworks: migrated (0.0025s) 

== 20210706142819 ReScheduleLatestPipelineIdPopulation: migrating =============
== 20210706142819 ReScheduleLatestPipelineIdPopulation: migrated (0.0000s) ====

== 20210706151154 AddRemainingCiMinutesToCiPendingBuild: migrating ============
-- add_column(:ci_pending_builds, :namespace_id, :bigint)
   -> 0.0007s
-- add_column(:ci_pending_builds, :minutes_exceeded, :boolean, {:null=>false, :default=>false})
   -> 0.0008s
== 20210706151154 AddRemainingCiMinutesToCiPendingBuild: migrated (0.0052s) ===

== 20210706151446 AddNamespaceForeignKeyToCiPendingBuild: migrating ===========
-- transaction_open?()
   -> 0.0000s
-- index_exists?(:ci_pending_builds, :namespace_id, {:name=>"index_ci_pending_builds_on_namespace_id", :algorithm=>:concurrently})
   -> 0.0034s
-- execute("SET statement_timeout TO 0")
   -> 0.0002s
-- add_index(:ci_pending_builds, :namespace_id, {:name=>"index_ci_pending_builds_on_namespace_id", :algorithm=>:concurrently})
   -> 0.0037s
-- execute("RESET statement_timeout")
   -> 0.0003s
-- transaction_open?()
   -> 0.0000s
-- foreign_keys(:ci_pending_builds)
   -> 0.0086s
-- execute("ALTER TABLE ci_pending_builds\nADD CONSTRAINT fk_fdc0137e4a\nFOREIGN KEY (namespace_id)\nREFERENCES namespaces (id)\nON DELETE CASCADE\nNOT VALID;\n")
   -> 0.0018s
-- execute("SET statement_timeout TO 0")
   -> 0.0002s
-- execute("ALTER TABLE ci_pending_builds VALIDATE CONSTRAINT fk_fdc0137e4a;")
   -> 0.0097s
-- execute("RESET statement_timeout")
   -> 0.0004s
== 20210706151446 AddNamespaceForeignKeyToCiPendingBuild: migrated (0.0328s) ==

== 20210706152139 AddIndexTypeToPostgresIndexesView: migrating ================
-- execute("DROP VIEW IF EXISTS postgres_indexes;\n\nCREATE VIEW postgres_indexes AS\nSELECT (pg_namespace.nspname::text || '.'::text) || i.relname::text AS identifier,\n  pg_index.indexrelid,\n  pg_namespace.nspname AS schema,\n  i.relname AS name,\n  pg_indexes.tablename,\n  a.amname AS type,\n  pg_index.indisunique AS \"unique\",\n  pg_index.indisvalid AS valid_index,\n  i.relispartition AS partitioned,\n  pg_index.indisexclusion AS exclusion,\n  pg_index.indexprs IS NOT NULL AS expression,\n  pg_index.indpred IS NOT NULL AS partial,\n  pg_indexes.indexdef AS definition,\n  pg_relation_size(i.oid::regclass) AS ondisk_size_bytes\nFROM pg_index\n  JOIN pg_class i ON i.oid = pg_index.indexrelid\n  JOIN pg_namespace ON i.relnamespace = pg_namespace.oid\n  JOIN pg_indexes ON i.relname = pg_indexes.indexname\n  JOIN pg_am a ON i.relam = a.oid\nWHERE pg_namespace.nspname <> 'pg_catalog'::name AND (pg_namespace.nspname = ANY (ARRAY[\"current_schema\"(), 'gitlab_partitions_dynamic'::name, 'gitlab_partitions_static'::name]));\n")
   -> 0.0091s
== 20210706152139 AddIndexTypeToPostgresIndexesView: migrated (0.0092s) =======

== 20210706212710 FinalizeCiJobArtifactsBigintConversion: migrating ===========
-- transaction_open?()
   -> 0.0000s
-- index_exists?("ci_job_artifacts", :id_convert_to_bigint, {:unique=>true, :name=>"index_ci_job_artifact_on_id_convert_to_bigint", :algorithm=>:concurrently})
   -> 0.0065s
-- execute("SET statement_timeout TO 0")
   -> 0.0003s
-- add_index("ci_job_artifacts", :id_convert_to_bigint, {:unique=>true, :name=>"index_ci_job_artifact_on_id_convert_to_bigint", :algorithm=>:concurrently})
   -> 0.0036s
-- execute("RESET statement_timeout")
   -> 0.0003s
-- transaction_open?()
   -> 0.0000s
-- index_exists?("ci_job_artifacts", [:project_id, :id_convert_to_bigint], {:name=>"index_ci_job_artifacts_for_terraform_reports_bigint", :where=>"file_type = 18", :algorithm=>:concurrently})
   -> 0.0054s
-- execute("SET statement_timeout TO 0")
   -> 0.0003s
-- add_index("ci_job_artifacts", [:project_id, :id_convert_to_bigint], {:name=>"index_ci_job_artifacts_for_terraform_reports_bigint", :where=>"file_type = 18", :algorithm=>:concurrently})
   -> 0.0041s
-- execute("RESET statement_timeout")
   -> 0.0003s
-- transaction_open?()
   -> 0.0000s
-- index_exists?("ci_job_artifacts", [:id_convert_to_bigint], {:name=>"index_ci_job_artifacts_id_for_terraform_reports_bigint", :where=>"file_type = 18", :algorithm=>:concurrently})
   -> 0.0054s
-- execute("SET statement_timeout TO 0")
   -> 0.0002s
-- add_index("ci_job_artifacts", [:id_convert_to_bigint], {:name=>"index_ci_job_artifacts_id_for_terraform_reports_bigint", :where=>"file_type = 18", :algorithm=>:concurrently})
   -> 0.0028s
-- execute("RESET statement_timeout")
   -> 0.0002s
-- transaction_open?()
   -> 0.0000s
-- foreign_keys(:project_pages_metadata)
   -> 0.0055s
-- execute("LOCK TABLE ci_job_artifacts, project_pages_metadata IN SHARE ROW EXCLUSIVE MODE")
   -> 0.0003s
-- execute("ALTER TABLE project_pages_metadata\nADD CONSTRAINT fk_69366a119e_tmp\nFOREIGN KEY (artifacts_archive_id)\nREFERENCES ci_job_artifacts (id_convert_to_bigint)\nON DELETE SET NULL\nNOT VALID;\n")
   -> 0.0011s
-- execute("SET statement_timeout TO 0")
   -> 0.0002s
-- execute("ALTER TABLE project_pages_metadata VALIDATE CONSTRAINT fk_69366a119e_tmp;")
   -> 0.0061s
-- execute("RESET statement_timeout")
   -> 0.0003s
-- execute("LOCK TABLE ci_job_artifacts, project_pages_metadata IN ACCESS EXCLUSIVE MODE")
   -> 0.0003s
-- quote_table_name("ci_job_artifacts")
   -> 0.0000s
-- quote_column_name(:id)
   -> 0.0000s
-- quote_column_name("id_tmp")
   -> 0.0000s
-- execute("ALTER TABLE \"ci_job_artifacts\" RENAME COLUMN \"id\" TO \"id_tmp\"")
   -> 0.0004s
-- quote_table_name("ci_job_artifacts")
   -> 0.0000s
-- quote_column_name(:id_convert_to_bigint)
   -> 0.0000s
-- quote_column_name(:id)
   -> 0.0000s
-- execute("ALTER TABLE \"ci_job_artifacts\" RENAME COLUMN \"id_convert_to_bigint\" TO \"id\"")
   -> 0.0004s
-- quote_table_name("ci_job_artifacts")
   -> 0.0000s
-- quote_column_name("id_tmp")
   -> 0.0000s
-- quote_column_name(:id_convert_to_bigint)
   -> 0.0000s
-- execute("ALTER TABLE \"ci_job_artifacts\" RENAME COLUMN \"id_tmp\" TO \"id_convert_to_bigint\"")
   -> 0.0003s
-- quote_table_name("trigger_be1804f21693")
   -> 0.0000s
-- execute("ALTER FUNCTION \"trigger_be1804f21693\" RESET ALL")
   -> 0.0003s
-- execute("ALTER SEQUENCE ci_job_artifacts_id_seq OWNED BY ci_job_artifacts.id")
   -> 0.0014s
-- change_column_default("ci_job_artifacts", :id, #<Proc:0x00007fc57f148bf0 /opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20210706212710_finalize_ci_job_artifacts_bigint_conversion.rb:66 (lambda)>)
   -> 0.0032s
-- change_column_default("ci_job_artifacts", :id_convert_to_bigint, 0)
   -> 0.0028s
-- execute("ALTER TABLE ci_job_artifacts DROP CONSTRAINT ci_job_artifacts_pkey CASCADE")
   -> 0.0033s
-- rename_index("ci_job_artifacts", "index_ci_job_artifact_on_id_convert_to_bigint", "ci_job_artifacts_pkey")
   -> 0.0004s
-- execute("ALTER TABLE ci_job_artifacts ADD CONSTRAINT ci_job_artifacts_pkey PRIMARY KEY USING INDEX ci_job_artifacts_pkey")
   -> 0.0006s
-- execute("DROP INDEX index_ci_job_artifacts_for_terraform_reports")
   -> 0.0009s
-- rename_index("ci_job_artifacts", "index_ci_job_artifacts_for_terraform_reports_bigint", "index_ci_job_artifacts_for_terraform_reports")
   -> 0.0004s
-- execute("DROP INDEX index_ci_job_artifacts_id_for_terraform_reports")
   -> 0.0006s
-- rename_index("ci_job_artifacts", "index_ci_job_artifacts_id_for_terraform_reports_bigint", "index_ci_job_artifacts_id_for_terraform_reports")
   -> 0.0004s
-- quote_table_name(:project_pages_metadata)
   -> 0.0000s
-- quote_column_name("fk_69366a119e_tmp")
   -> 0.0000s
-- quote_column_name("fk_69366a119e")
   -> 0.0000s
-- execute("ALTER TABLE \"project_pages_metadata\"\nRENAME CONSTRAINT \"fk_69366a119e_tmp\" TO \"fk_69366a119e\"\n")
   -> 0.0004s
== 20210706212710 FinalizeCiJobArtifactsBigintConversion: migrated (0.0736s) ==

== 20210706213537 AddPremiumAndUltimatePlanLimits: migrating ==================
== 20210706213537 AddPremiumAndUltimatePlanLimits: migrated (0.0000s) =========

== 20210707095545 AddStatusToMergeRequestCleanupSchedules: migrating ==========
-- column_exists?(:merge_request_cleanup_schedules, :status)
   -> 0.0013s
-- add_column(:merge_request_cleanup_schedules, :status, :integer, {:limit=>2, :default=>0, :null=>false})
   -> 0.0026s
-- transaction_open?()
   -> 0.0000s
-- index_exists?(:merge_request_cleanup_schedules, :status, {:name=>"index_merge_request_cleanup_schedules_on_status", :algorithm=>:concurrently})
   -> 0.0019s
-- execute("SET statement_timeout TO 0")
   -> 0.0003s
-- add_index(:merge_request_cleanup_schedules, :status, {:name=>"index_merge_request_cleanup_schedules_on_status", :algorithm=>:concurrently})
   -> 0.0032s
-- execute("RESET statement_timeout")
   -> 0.0003s
== 20210707095545 AddStatusToMergeRequestCleanupSchedules: migrated (0.0107s) =

== 20210707113056 AddTagsArrayToCiPendingBuilds: migrating ====================
-- add_column(:ci_pending_builds, :tag_ids, :integer, {:array=>true, :default=>[]})
   -> 0.0032s
== 20210707113056 AddTagsArrayToCiPendingBuilds: migrated (0.0050s) ===========

== 20210707151536 CreateVulnerabilityFindingEvidenceSources: migrating ========
-- create_table(:vulnerability_finding_evidence_sources, {})
-- quote_column_name(:name)
   -> 0.0000s
-- quote_column_name(:url)
   -> 0.0000s
   -> 0.0077s
-- quote_table_name("check_86b537ba1a")
   -> 0.0000s
-- quote_table_name("check_0fe01298d6")
   -> 0.0000s
-- quote_table_name(:vulnerability_finding_evidence_sources)
   -> 0.0000s
-- execute("ALTER TABLE \"vulnerability_finding_evidence_sources\"\nADD CONSTRAINT \"check_86b537ba1a\" CHECK (char_length(\"name\") <= 2048),\nADD CONSTRAINT \"check_0fe01298d6\" CHECK (char_length(\"url\") <= 2048)\n")rake aborted!
StandardError: An error has occurred, all later migrations canceled:

Expected batched background migration for the given configuration to be marked as 'finished', but it is 'active':	{:job_class_name=>"CopyColumnUsingBackgroundMigrationJob", :table_name=>"ci_stages", :column_name=>"id", :job_arguments=>[["id"], ["id_convert_to_bigint"]]}

Finalize it manualy by running

	sudo gitlab-rake gitlab:background_migrations:finalize[CopyColumnUsingBackgroundMigrationJob,ci_stages,id,'[["id"]\, ["id_convert_to_bigint"]]']

For more information, check the documentation

	https://docs.gitlab.com/ee/user/admin_area/monitoring/background_migrations.html#database-migrations-failing-because-of-batched-background-migration-not-finished
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:1109:in `ensure_batched_background_migration_is_finished'
/opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20210707210916_finalize_ci_stages_bigint_conversion.rb:13:in `up'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/lock_retry_mixin.rb:31:in `ddl_transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:61:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'

Caused by:
Expected batched background migration for the given configuration to be marked as 'finished', but it is 'active':	{:job_class_name=>"CopyColumnUsingBackgroundMigrationJob", :table_name=>"ci_stages", :column_name=>"id", :job_arguments=>[["id"], ["id_convert_to_bigint"]]}

Finalize it manualy by running

	sudo gitlab-rake gitlab:background_migrations:finalize[CopyColumnUsingBackgroundMigrationJob,ci_stages,id,'[["id"]\, ["id_convert_to_bigint"]]']

For more information, check the documentation

	https://docs.gitlab.com/ee/user/admin_area/monitoring/background_migrations.html#database-migrations-failing-because-of-batched-background-migration-not-finished
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:1109:in `ensure_batched_background_migration_is_finished'
/opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20210707210916_finalize_ci_stages_bigint_conversion.rb:13:in `up'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/lock_retry_mixin.rb:31:in `ddl_transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:61:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Tasks: TOP => db:migrate
(See full trace by running task with --trace)

   -> 0.0008s
== 20210707151536 CreateVulnerabilityFindingEvidenceSources: migrated (0.0123s) 

== 20210707163659 AddVulnerabilityEventsToIntegrations: migrating =============
-- add_column(:integrations, :vulnerability_events, :boolean, {:default=>false, :null=>false})
   -> 0.0020s
== 20210707163659 AddVulnerabilityEventsToIntegrations: migrated (0.0020s) ====

== 20210707171536 CreateVulnerabilityFindingEvidenceAssets: migrating =========
-- create_table(:vulnerability_finding_evidence_assets, {})
-- quote_column_name(:type)
   -> 0.0000s
-- quote_column_name(:name)
   -> 0.0000s
-- quote_column_name(:url)
   -> 0.0000s
   -> 0.0075s
-- quote_table_name("check_5adf5d69de")
   -> 0.0000s
-- quote_table_name("check_839f29d7ca")
   -> 0.0000s
-- quote_table_name("check_9272d912c0")
   -> 0.0000s
-- quote_table_name(:vulnerability_finding_evidence_assets)
   -> 0.0000s
-- execute("ALTER TABLE \"vulnerability_finding_evidence_assets\"\nADD CONSTRAINT \"check_5adf5d69de\" CHECK (char_length(\"type\") <= 2048),\nADD CONSTRAINT \"check_839f29d7ca\" CHECK (char_length(\"name\") <= 2048),\nADD CONSTRAINT \"check_9272d912c0\" CHECK (char_length(\"url\") <= 2048)\n")
   -> 0.0006s
== 20210707171536 CreateVulnerabilityFindingEvidenceAssets: migrated (0.0115s) 

== 20210707171554 CreateVulnerabilityFlags: migrating =========================
-- create_table(:vulnerability_flags, {})
-- quote_column_name(:origin)
   -> 0.0000s
-- quote_column_name(:description)
   -> 0.0000s
   -> 0.0087s
-- quote_table_name("check_49c1d00032")
   -> 0.0000s
-- quote_table_name("check_45e743349f")
   -> 0.0000s
-- quote_table_name(:vulnerability_flags)
   -> 0.0000s
-- execute("ALTER TABLE \"vulnerability_flags\"\nADD CONSTRAINT \"check_49c1d00032\" CHECK (char_length(\"origin\") <= 255),\nADD CONSTRAINT \"check_45e743349f\" CHECK (char_length(\"description\") <= 1024)\n")
   -> 0.0006s
== 20210707171554 CreateVulnerabilityFlags: migrated (0.0127s) ================

== 20210707173645 AddProjectSettingsPreviousDefaultBranchTextLimit: migrating =
-- transaction_open?()
   -> 0.0000s
-- current_schema()
   -> 0.0003s
-- execute("ALTER TABLE project_settings\nADD CONSTRAINT check_3a03e7557a\nCHECK ( char_length(previous_default_branch) <= 4096 )\nNOT VALID;\n")
   -> 0.0005s
-- current_schema()
   -> 0.0003s
-- execute("SET statement_timeout TO 0")
   -> 0.0003s
-- execute("ALTER TABLE project_settings VALIDATE CONSTRAINT check_3a03e7557a;")
   -> 0.0014s
-- execute("RESET statement_timeout")
   -> 0.0002s
== 20210707173645 AddProjectSettingsPreviousDefaultBranchTextLimit: migrated (0.0077s) 

== 20210707181536 CreateVulnerabilityFindingEvidenceSupportingMessages: migrating 
-- create_table(:vulnerability_finding_evidence_supporting_messages, {})
-- quote_column_name(:name)
   -> 0.0000s
   -> 0.0071s
-- quote_table_name("check_fa33b9ae85")
   -> 0.0000s
-- quote_table_name(:vulnerability_finding_evidence_supporting_messages)
   -> 0.0000s
-- execute("ALTER TABLE \"vulnerability_finding_evidence_supporting_messages\"\nADD CONSTRAINT \"check_fa33b9ae85\" CHECK (char_length(\"name\") <= 2048)\n")
   -> 0.0004s
== 20210707181536 CreateVulnerabilityFindingEvidenceSupportingMessages: migrated (0.0107s) 

== 20210707210916 FinalizeCiStagesBigintConversion: migrating =================
STDERR:
---- End output of "bash"  "/tmp/chef-script20211005-2430-18bo7hn" ----
Ran "bash"  "/tmp/chef-script20211005-2430-18bo7hn" returned 1

Running sudo gitlab-rake gitlab:background_migrations:finalize[CopyColumnUsingBackgroundMigrationJob,ci_builds,id,'[["id"\, "stage_id"]\, ["id_convert_to_bigint"\, "stage_id_convert_to_bigint"]]'] and then reconfiguring also doesn’t work.

Trying to dowdgrade to 14.0 again with sudo apt install gitlab-ee=14.0.* gives me following error:

Reading package lists... Done
Building dependency tree       
Reading state information... Done
Selected version '14.0.11-ee.0' (gitlab-ee:1/bionic [amd64]) for 'gitlab-ee'
The following packages will be DOWNGRADED:
  gitlab-ee
0 upgraded, 0 newly installed, 1 downgraded, 0 to remove and 0 not upgraded.
Need to get 0 B/990 MB of archives.
After this operation, 31.7 MB disk space will be freed.
Do you want to continue? [Y/n] Y
dpkg: warning: downgrading gitlab-ee from 14.3.2-ee.0 to 14.0.11-ee.0
(Reading database ... 237357 files and directories currently installed.)
Preparing to unpack .../gitlab-ee_14.0.11-ee.0_amd64.deb ...
Malformed configuration JSON file found at /opt/gitlab/embedded/nodes/gitlab.json.
This usually happens when your last run of `gitlab-ctl reconfigure` didn't complete successfully.
This file is used to check if any of the unsupported configurations are enabled,
and hence require a working reconfigure before upgrading.
Please run `sudo gitlab-ctl reconfigure` to fix it and try again.
dpkg: error processing archive /var/cache/apt/archives/gitlab-ee_14.0.11-ee.0_amd64.deb (--unpack):
 new gitlab-ee package pre-installation script subprocess returned error exit status 1
Errors were encountered while processing:
 /var/cache/apt/archives/gitlab-ee_14.0.11-ee.0_amd64.deb
E: Sub-process /usr/bin/dpkg returned an error code (1)

Now I’m stuck and can’t use the gitlab instance. This is quite bad and I’m really wondering why this is happening because I only use the official recommended deb packages.

Has anyone else this issue?

Yep, I am having the same issue.

Though I have this problem using docker.

Caused by:
Expected batched background migration for the given configuration to be marked as ‘finished’, but it is ‘active’: {
:job_class_name=>“CopyColumnUsingBackgroundMigrationJob”, :table_name=>“ci_builds”, :column_name=>“id”, :job_argume
nts=>[[“id”, “stage_id”], [“id_convert_to_bigint”, “stage_id_convert_to_bigint”]]}
Finalize it manualy by running
sudo gitlab-rake gitlab:background_migrations:finalize[CopyColumnUsingBackgroundMigrationJob,ci_builds,id,’
[[“id”, “stage_id”], [“id_convert_to_bigint”, “stage_id_convert_to_bigint”]]’]
For more information, check the documentation

That command says done, but gitlab never finishes the job.

Just got the same error while upgrading 14.0.11 to the latest release on CentOS 7.

Reverting to an older version doesn’t seem to work.

Same error on Ubuntu 18. Anyone able to fix?

This fixed it for me on Amazon Linux 2:

sudo gitlab-rake gitlab:background_migrations:finalize[CopyColumnUsingBackgroundMigrationJob,ci_stages,id,’[[“id”], [“id_convert_to_bigint”]]’]

sudo gitlab-rake gitlab:background_migrations:finalize[CopyColumnUsingBackgroundMigrationJob,push_event_payloads,event_id,’[[“event_id”], [“event_id_convert_to_bigint”]]’]

sudo gitlab-ctl reconfigure

More info here: Batched background migrations | GitLab

I finally did it! @trevorjohnson commands didn’t work for me, but after I saw the link I kept digging and found this: Upgrading GitLab | GitLab
So I ran:

sudo gitlab-rake db:migrate
sudo gitlab-ctl reconfigure
sudo gitlab-ctl restart

Now it seems to be working fine. Thanks for the hint!

We did something similar to LFlo after a few rounds of attempting to recomplete the upgrade on ubuntu server.

We were confused because checking the remaining processes with the following command resulted in 0:

sudo gitlab-rails runner -e production 'puts Gitlab::BackgroundMigration.remaining'

Using the other combinations of commands didn’t seem to help, but this in exactly this order did solve it for us:

sudo gitlab-rake gitlab:background_migrations:finalize[CopyColumnUsingBackgroundMigrationJob,push_event_payloads,event_id,'[["event_id"]\, ["event_id_convert_to_bigint"]]']
sudo gitlab-ctl reconfigure
sudo gitlab-ctl restart

We think @LFlo may have the better commands overall but we solved the problem just before his post was up, so didn’t get to try it.

Thank you to everyone on this thread. We would not have solved this without you and now we have more time to search for the appropriate replacement to Gitlab. Issues like these have caused us to look elsewhere.

需要在14.2.6 或者14.2.7 上版本上执行这些语句，才可以升级到下一个版本