Upgrading From 14.4.5 upwards

How to Use GitLab Upgrade
1

Hi Please Asist, I tried literally anything!
I am trying to upgrade my gitlab from version 14.4.5 onwards (14.5.0 up to latest)
and the update allways fails.
I am using gitlab-omnibus installation over regular Centos7.latest not a container.
I am updating via official gitlab repo with Yum.

what I tried:

  1. Droping indices related to the error.

  2. gitlab-rake db:migrate.

  3. any rake script that I have found in the forum someone provided and might be even remotely related (check-out a post I have written before, Upgrading to anything above 14.4.5 fails - #2 by iwalker in which I think no body have seen my follow ups)

  4. gitlab-rake gitlab:check - always fine:

  5. gitlab-rake gitlab:env:info:

here is the error from the installation output:

rails_migration[gitlab-rails] (gitlab::database_migrations line 51) had an error: Mixlib::ShellOut::CommandTimeout: bash[migrate gitlab-rails database] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/resources/rails_migration.rb line 16) had an error: Mixlib::ShellOut::CommandTimeout: Command timed out after 3600s:
Command exceeded allowed execution time, process terminated
---- Begin output of “bash” “/tmp/chef-script20220322-14277-1llxqe6” ----
STDOUT: WARNING: Active Record does not support composite primary key.

project_authorizations has composite primary key. Composite primary key is ignored.
== 20210713042000 FixCiSourcesPipelinesIndexNames: migrating ==================
– foreign_keys(“ci_sources_pipelines”)
→ 0.0139s
– foreign_keys(“ci_sources_pipelines”)
→ 0.0057s
– foreign_keys(“ci_sources_pipelines”)
→ 0.0051s
– foreign_keys(“ci_sources_pipelines”)
→ 0.0051s
– foreign_keys(“ci_sources_pipelines”)
→ 0.0051s
– indexes(“ci_sources_pipelines”)
→ 0.0075s
– current_schema()
→ 0.0006s
== 20210713042000 FixCiSourcesPipelinesIndexNames: migrated (0.0495s) =========

== 20210811193033 AddUniqueIndexToVulnerabilityFindingLinks: migrating ========
== 20210811193033 AddUniqueIndexToVulnerabilityFindingLinks: migrated (0.0000s)

== 20210812013042 RemoveDuplicateProjectAuthorizations: migrating =============
– transaction_open?()
→ 0.0000s
– index_exists?(:project_authorizations, [:project_id, :user_id], {:unique=>true, :name=>“index_unique_project_authorizations_on_project_id_user_id”, :algorithm=>:concurrently})
→ 0.0024s
– execute(“SET statement_timeout TO 0”)
→ 0.0005s
– add_index(:project_authorizations, [:project_id, :user_id], {:unique=>true, :name=>“index_unique_project_authorizations_on_project_id_user_id”, :algorithm=>:concurrently})
→ 0.0264s
– execute(“RESET statement_timeout”)
→ 0.0007s
– transaction_open?()
→ 0.0000s
– indexes(:project_authorizations)
→ 0.0030s
– execute(“SET statement_timeout TO 0”)
→ 0.0005s
– remove_index(:project_authorizations, {:algorithm=>:concurrently, :name=>“index_project_authorizations_on_project_id_user_id”})
→ 0.0586s
– execute(“RESET statement_timeout”)
→ 0.0008s
== 20210812013042 RemoveDuplicateProjectAuthorizations: migrated (0.3349s) ====

== 20210923192648 RemoveForeignKeysFromOpenProjectDataTable: migrating ========
– transaction_open?()
→ 0.0000s
– remove_foreign_key(:open_project_tracker_data, {:column=>:service_id})
→ 0.0084s
== 20210923192648 RemoveForeignKeysFromOpenProjectDataTable: migrated (0.0216s)

== 20210923192649 RemoveOpenProjectDataTable: migrating =======================
– drop_table(:open_project_tracker_data)
→ 0.0102s
== 20210923192649 RemoveOpenProjectDataTable: migrated (0.0102s) ==============

== 20210930211936 BackfillUserNamespace: migrating ============================
== 20210930211936 BackfillUserNamespace: migrated (0.1041s) ===================

== 20211004120135 ScheduleAddPrimaryEmailToEmailsIfUserConfirmed: migrating ===
– Scheduled 1 AddPrimaryEmailToEmailsIfUserConfirmed jobs with a maximum of 10000 records per batch and an interval of 120 seconds.

The migration is expected to take at least 120 seconds. Expect all jobs to have completed after 2022-03-22 20:00:48 UTC."
== 20211004120135 ScheduleAddPrimaryEmailToEmailsIfUserConfirmed: migrated (0.2497s)

== 20211004122540 CreateMemberTasks: migrating ================================
– create_table(:member_tasks, {})
→ 0.0188s
== 20211004122540 CreateMemberTasks: migrated (0.0188s) =======================

== 20211005112404 AddMemberIdForeignKeyToMemberTasks: migrating ===============
– transaction_open?()
→ 0.0000s
– foreign_keys(:member_tasks)
→ 0.0053s
– transaction_open?()
→ 0.0000s
– execute(“ALTER TABLE member_tasks\nADD CONSTRAINT fk_12816d4bbb\nFOREIGN KEY (member_id)\nREFERENCES members (id)\nON DELETE CASCADE\nNOT VALID;\n”)
→ 0.0047s
– execute(“SET statement_timeout TO 0”)
→ 0.0006s
– execute(“ALTER TABLE member_tasks VALIDATE CONSTRAINT fk_12816d4bbb;”)
→ 0.0210s
– execute(“RESET statement_timeout”)
→ 0.0007s
== 20211005112404 AddMemberIdForeignKeyToMemberTasks: migrated (0.0642s) ======

== 20211005112645 AddProjectIdForeignKeyToMemberTasks: migrating ==============
– transaction_open?()
→ 0.0000s
– foreign_keys(:member_tasks)
→ 0.0122s
– transaction_open?()
→ 0.0000s
– execute(“ALTER TABLE member_tasks\nADD CONSTRAINT fk_ab636303dd\nFOREIGN KEY (project_id)\nREFERENCES projects (id)\nON DELETE CASCADE\nNOT VALID;\n”)
→ 0.0041s
– execute(“SET statement_timeout TO 0”)
→ 0.0004s
– execute(“ALTER TABLE member_tasks VALIDATE CONSTRAINT fk_ab636303dd;”)
→ 0.0049s
– execute(“RESET statement_timeout”)
→ 0.0005s
== 20211005112645 AddProjectIdForeignKeyToMemberTasks: migrated (0.0287s) =====

== 20211005194425 ScheduleRequirementsMigration: migrating ====================
– Scheduled 0 MigrateRequirementsToWorkItems jobs with a maximum of 30 records per batch and an interval of 120 seconds.

The migration is expected to take at least 0 seconds. Expect all jobs to have completed after 2022-03-22 19:58:49 UTC."
== 20211005194425 ScheduleRequirementsMigration: migrated (0.0861s) ===========

== 20211007090229 CreateIssueSearchTable: migrating ===========================
– execute(“CREATE TABLE issue_search_data (\n project_id bigint NOT NULL REFERENCES projects(id) ON DELETE CASCADE,\n issue_id bigint NOT NULL REFERENCES issues(id) ON DELETE CASCADE,\n created_at timestamp with time zone DEFAULT NOW() NOT NULL,\n updated_at timestamp with time zone DEFAULT NOW() NOT NULL,\n search_vector tsvector,\n PRIMARY KEY (project_id, issue_id)\n) PARTITION BY HASH (project_id)\n”)
→ 0.0085s
– add_index(:issue_search_data, :issue_id)
→ 0.0009s
– add_index(:issue_search_data, :search_vector, {:using=>:gin, :name=>“index_issue_search_data_on_search_vector”})
→ 0.0008s
– transaction()
– execute(“CREATE TABLE gitlab_partitions_static.issue_search_data_00\nPARTITION OF issue_search_data\nFOR VALUES WITH (MODULUS 64, REMAINDER 0);\n”)
→ 0.0096s
– execute(“CREATE TABLE gitlab_partitions_static.issue_search_data_01\nPARTITION OF issue_search_data\nFOR VALUES WITH (MODULUS 64, REMAINDER 1);\n”)
→ 0.0039s
– execute(“CREATE TABLE gitlab_partitions_static.issue_search_data_02\nPARTITION OF issue_search_data\nFOR VALUES WITH (MODULUS 64, REMAINDER 2);\n”)
→ 0.0036s
– execute(“CREATE TABLE gitlab_partitions_static.issue_search_data_03\nPARTITION OF issue_search_data\nFOR VALUES WITH (MODULUS 64, REMAINDER 3);\n”)
→ 0.0033s
– execute(“CREATE TABLE gitlab_partitions_static.issue_search_data_04\nPARTITION OF issue_search_data\nFOR VALUES WITH (MODULUS 64, REMAINDER 4);\n”)
→ 0.0032s
– execute(“CREATE TABLE gitlab_partitions_static.issue_search_data_05\nPARTITION OF issue_search_data\nFOR VALUES WITH (MODULUS 64, REMAINDER 5);\n”)
→ 0.0114s
– execute(“CREATE TABLE gitlab_partitions_static.issue_search_data_06\nPARTITION OF issue_search_data\nFOR VALUES WITH (MODULUS 64, REMAINDER 6);\n”)
→ 0.0068s
– execute(“CREATE TABLE gitlab_partitions_static.issue_search_data_07\nPARTITION OF issue_search_data\nFOR VALUES WITH (MODULUS 64, REMAINDER 7);\n”)
→ 0.0048s
– execute(“CREATE TABLE gitlab_partitions_static.issue_search_data_08\nPARTITION OF issue_search_data\nFOR VALUES WITH (MODULUS 64, REMAINDER 8);\n”)
→ 0.0035s
– execute(“CREATE TABLE gitlab_partitions_static.issue_search_data_09\nPARTITION OF issue_search_data\nFOR VALUES WITH (MODULUS 64, REMAINDER 9);\n”)
→ 0.0035s
– execute(“CREATE TABLE gitlab_partitions_static.issue_search_data_10\nPARTITION OF issue_search_data\nFOR VALUES WITH (MODULUS 64, REMAINDER 10);\n”)
→ 0.0033s
– execute(“CREATE TABLE gitlab_partitions_static.issue_search_data_11\nPARTITION OF issue_search_data\nFOR VALUES WITH (MODULUS 64, REMAINDER 11);\n”)
→ 0.0051s
– execute(“CREATE TABLE gitlab_partitions_static.issue_search_data_12\nPARTITION OF issue_search_data\nFOR VALUES WITH (MODULUS 64, REMAINDER 12);\n”)
→ 0.0032s
– execute(“CREATE TABLE gitlab_partitions_static.issue_search_data_13\nPARTITION OF issue_search_data\nFOR VALUES WITH (MODULUS 64, REMAINDER 13);\n”)
→ 0.0031s
– execute(“CREATE TABLE gitlab_partitions_static.issue_search_data_14\nPARTITION OF issue_search_data\nFOR VALUES WITH (MODULUS 64, REMAINDER 14);\n”)
→ 0.0033s
– execute(“CREATE TABLE gitlab_partitions_static.issue_search_data_15\nPARTITION OF issue_search_data\nFOR VALUES WITH (MODULUS 64, REMAINDER 15);\n”)
→ 0.0031s
– execute(“CREATE TABLE gitlab_partitions_static.issue_search_data_16\nPARTITION OF issue_search_data\nFOR VALUES WITH (MODULUS 64, REMAINDER 16);\n”)
→ 0.0035s
– execute(“CREATE TABLE gitlab_partitions_static.issue_search_data_17\nPARTITION OF issue_search_data\nFOR VALUES WITH (MODULUS 64, REMAINDER 17);\n”)
→ 0.0032s
– execute(“CREATE TABLE gitlab_partitions_static.issue_search_data_18\nPARTITION OF issue_search_data\nFOR VALUES WITH (MODULUS 64, REMAINDER 18);\n”)
→ 0.0033s
– execute(“CREATE TABLE gitlab_partitions_static.issue_search_data_19\nPARTITION OF issue_search_data\nFOR VALUES WITH (MODULUS 64, REMAINDER 19);\n”)
→ 0.0032s
– execute(“CREATE TABLE gitlab_partitions_static.issue_search_data_20\nPARTITION OF issue_search_data\nFOR VALUES WITH (MODULUS 64, REMAINDER 20);\n”)
→ 0.0034s
– execute(“CREATE TABLE gitlab_partitions_static.issue_search_data_21\nPARTITION OF issue_search_data\nFOR VALUES WITH (MODULUS 64, REMAINDER 21);\n”)
→ 0.0035s
– execute(“CREATE TABLE gitlab_partitions_static.issue_search_data_22\nPARTITION OF issue_search_data\nFOR VALUES WITH (MODULUS 64, REMAINDER 22);\n”)
→ 0.0034s
– execute(“CREATE TABLE gitlab_partitions_static.issue_search_data_23\nPARTITION OF issue_search_data\nFOR VALUES WITH (MODULUS 64, REMAINDER 23);\n”)
→ 0.0033s
– execute(“CREATE TABLE gitlab_partitions_static.issue_search_data_24\nPARTITION OF issue_search_data\nFOR VALUES WITH (MODULUS 64, REMAINDER 24);\n”)
→ 0.0034s
– execute(“CREATE TABLE gitlab_partitions_static.issue_search_data_25\nPARTITION OF issue_search_data\nFOR VALUES WITH (MODULUS 64, REMAINDER 25);\n”)
→ 0.0032s
– execute(“CREATE TABLE gitlab_partitions_static.issue_search_data_26\nPARTITION OF issue_search_data\nFOR VALUES WITH (MODULUS 64, REMAINDER 26);\n”)
→ 0.0033s
– execute(“CREATE TABLE gitlab_partitions_static.issue_search_data_27\nPARTITION OF issue_search_data\nFOR VALUES WITH (MODULUS 64, REMAINDER 27);\n”)
→ 0.0034s
– execute(“CREATE TABLE gitlab_partitions_static.issue_search_data_28\nPARTITION OF issue_search_data\nFOR VALUES WITH (MODULUS 64, REMAINDER 28);\n”)
→ 0.0034s
– execute(“CREATE TABLE gitlab_partitions_static.issue_search_data_29\nPARTITION OF issue_search_data\nFOR VALUES WITH (MODULUS 64, REMAINDER 29);\n”)
→ 0.0034s
– execute(“CREATE TABLE gitlab_partitions_static.issue_search_data_30\nPARTITION OF issue_search_data\nFOR VALUES WITH (MODULUS 64, REMAINDER 30);\n”)
→ 0.0034s
– execute(“CREATE TABLE gitlab_partitions_static.issue_search_data_31\nPARTITION OF issue_search_data\nFOR VALUES WITH (MODULUS 64, REMAINDER 31);\n”)
→ 0.0035s
– execute(“CREATE TABLE gitlab_partitions_static.issue_search_data_32\nPARTITION OF issue_search_data\nFOR VALUES WITH (MODULUS 64, REMAINDER 32);\n”)
→ 0.0033s
– execute(“CREATE TABLE gitlab_partitions_static.issue_search_data_33\nPARTITION OF issue_search_data\nFOR VALUES WITH (MODULUS 64, REMAINDER 33);\n”)
→ 0.0031s
– execute(“CREATE TABLE gitlab_partitions_static.issue_search_data_34\nPARTITION OF issue_search_data\nFOR VALUES WITH (MODULUS 64, REMAINDER 34);\n”)
→ 0.0032s
– execute(“CREATE TABLE gitlab_partitions_static.issue_search_data_35\nPARTITION OF issue_search_data\nFOR VALUES WITH (MODULUS 64, REMAINDER 35);\n”)
→ 0.0267s
– execute(“CREATE TABLE gitlab_partitions_static.issue_search_data_36\nPARTITION OF issue_search_data\nFOR VALUES WITH (MODULUS 64, REMAINDER 36);\n”)
→ 0.0112s
– execute(“CREATE TABLE gitlab_partitions_static.issue_search_data_37\nPARTITION OF issue_search_data\nFOR VALUES WITH (MODULUS 64, REMAINDER 37);\n”)
→ 0.0061s
– execute(“CREATE TABLE gitlab_partitions_static.issue_search_data_38\nPARTITION OF issue_search_data\nFOR VALUES WITH (MODULUS 64, REMAINDER 38);\n”)
→ 0.0084s
– execute(“CREATE TABLE gitlab_partitions_static.issue_search_data_39\nPARTITION OF issue_search_data\nFOR VALUES WITH (MODULUS 64, REMAINDER 39);\n”)
→ 0.0081s
– execute(“CREATE TABLE gitlab_partitions_static.issue_search_data_40\nPARTITION OF issue_search_data\nFOR VALUES WITH (MODULUS 64, REMAINDER 40);\n”)
→ 0.0102s
– execute(“CREATE TABLE gitlab_partitions_static.issue_search_data_41\nPARTITION OF issue_search_data\nFOR VALUES WITH (MODULUS 64, REMAINDER 41);\n”)
→ 0.0082s
– execute(“CREATE TABLE gitlab_partitions_static.issue_search_data_42\nPARTITION OF issue_search_data\nFOR VALUES WITH (MODULUS 64, REMAINDER 42);\n”)
→ 0.0053s
– execute(“CREATE TABLE gitlab_partitions_static.issue_search_data_43\nPARTITION OF issue_search_data\nFOR VALUES WITH (MODULUS 64, REMAINDER 43);\n”)
→ 0.0092s
– execute(“CREATE TABLE gitlab_partitions_static.issue_search_data_44\nPARTITION OF issue_search_data\nFOR VALUES WITH (MODULUS 64, REMAINDER 44);\n”)
→ 0.0071s
– execute(“CREATE TABLE gitlab_partitions_static.issue_search_data_45\nPARTITION OF issue_search_data\nFOR VALUES WITH (MODULUS 64, REMAINDER 45);\n”)
→ 0.0057s
– execute(“CREATE TABLE gitlab_partitions_static.issue_search_data_46\nPARTITION OF issue_search_data\nFOR VALUES WITH (MODULUS 64, REMAINDER 46);\n”)
→ 0.0047s
– execute(“CREATE TABLE gitlab_partitions_static.issue_search_data_47\nPARTITION OF issue_search_data\nFOR VALUES WITH (MODULUS 64, REMAINDER 47);\n”)
→ 0.0057s
– execute(“CREATE TABLE gitlab_partitions_static.issue_search_data_48\nPARTITION OF issue_search_data\nFOR VALUES WITH (MODULUS 64, REMAINDER 48);\n”)
→ 0.0108s
– execute(“CREATE TABLE gitlab_partitions_static.issue_search_data_49\nPARTITION OF issue_search_data\nFOR VALUES WITH (MODULUS 64, REMAINDER 49);\n”)
→ 0.0106s
– execute(“CREATE TABLE gitlab_partitions_static.issue_search_data_50\nPARTITION OF issue_search_data\nFOR VALUES WITH (MODULUS 64, REMAINDER 50);\n”)
→ 0.0065s
– execute(“CREATE TABLE gitlab_partitions_static.issue_search_data_51\nPARTITION OF issue_search_data\nFOR VALUES WITH (MODULUS 64, REMAINDER 51);\n”)
→ 0.0047s
– execute(“CREATE TABLE gitlab_partitions_static.issue_search_data_52\nPARTITION OF issue_search_data\nFOR VALUES WITH (MODULUS 64, REMAINDER 52);\n”)
→ 0.0040s
– execute(“CREATE TABLE gitlab_partitions_static.issue_search_data_53\nPARTITION OF issue_search_data\nFOR VALUES WITH (MODULUS 64, REMAINDER 53);\n”)
→ 0.0035s
– execute(“CREATE TABLE gitlab_partitions_static.issue_search_data_54\nPARTITION OF issue_search_data\nFOR VALUES WITH (MODULUS 64, REMAINDER 54);\n”)
→ 0.0037s
– execute(“CREATE TABLE gitlab_partitions_static.issue_search_data_55\nPARTITION OF issue_search_data\nFOR VALUES WITH (MODULUS 64, REMAINDER 55);\n”)
→ 0.0138s
– execute(“CREATE TABLE gitlab_partitions_static.issue_search_data_56\nPARTITION OF issue_search_data\nFOR VALUES WITH (MODULUS 64, REMAINDER 56);\n”)
→ 0.0195s
– execute(“CREATE TABLE gitlab_partitions_static.issue_search_data_57\nPARTITION OF issue_search_data\nFOR VALUES WITH (MODULUS 64, REMAINDER 57);\n”)
→ 0.0110s
– execute(“CREATE TABLE gitlab_partitions_static.issue_search_data_58\nPARTITION OF issue_search_data\nFOR VALUES WITH (MODULUS 64, REMAINDER 58);\n”)
→ 0.0139s
– execute(“CREATE TABLE gitlab_partitions_static.issue_search_data_59\nPARTITION OF issue_search_data\nFOR VALUES WITH (MODULUS 64, REMAINDER 59);\n”)
→ 0.0092s
– execute(“CREATE TABLE gitlab_partitions_static.issue_search_data_60\nPARTITION OF issue_search_data\nFOR VALUES WITH (MODULUS 64, REMAINDER 60);\n”)
→ 0.0052s
– execute(“CREATE TABLE gitlab_partitions_static.issue_search_data_61\nPARTITION OF issue_search_data\nFOR VALUES WITH (MODULUS 64, REMAINDER 61);\n”)
→ 0.0072s
– execute(“CREATE TABLE gitlab_partitions_static.issue_search_data_62\nPARTITION OF issue_search_data\nFOR VALUES WITH (MODULUS 64, REMAINDER 62);\n”)
→ 0.0091s
– execute(“CREATE TABLE gitlab_partitions_static.issue_search_data_63\nPARTITION OF issue_search_data\nFOR VALUES WITH (MODULUS 64, REMAINDER 63);\n”)
→ 0.0071s
→ 0.4021s
== 20211007090229 CreateIssueSearchTable: migrated (0.4125s) ==================

== 20211008043855 RemoveNotesTrigramIndex: migrating ==========================
– transaction_open?()
→ 0.0000s
– indexes(:notes)
→ 0.0130s
– execute(“SET statement_timeout TO 0”)
→ 0.0004s
– remove_index(:notes, {:algorithm=>:concurrently, :name=>“index_notes_on_note_trigram”})
STDERR:
---- End output of “bash” “/tmp/chef-script20220322-14277-1llxqe6” ----
Ran “bash” “/tmp/chef-script20220322-14277-1llxqe6” returned

Running handlers complete

Chef Infra Client failed. 53 resources updated in 01 hours 02 minutes 09 seconds

There was an error running gitlab-ctl reconfigure. Please check the output above for more

details.

warning: %posttrans(gitlab-ce-14.9.0-ce.0.el7.x86_64) scriptlet failed, exit status 1
Non-fatal POSTTRANS scriptlet failure in rpm package gitlab-ce-14.9.1-ce.0.el7.x86_64
Verifying : gitlab-ce-14.9.1-ce.0.el7.x86_64 1/2
Verifying : gitlab-ce-14.4.5-ce.0.el7.x86_64 2/2

Updated:
gitlab-ce.x86_64 0:14.9.1-ce.0.el7

Complete!
You have new mail in /var/spool/mail/root

after few month I am really desprate, please assit!