Unable to Upgrade from 16.1.5 to 16.2.#

General
1

Hey all,

I’ve been battling our in house Gitlab for a couple months. We appear to have the id_convert_to_bigint migration problem from 15.11.# here. We separated our DB in version 16.0.0 and upgraded to 16.1.5 with no apparent issues.

Note: I’ve been doing the fix before trying to upgrade. apply fix and wait for migrations to finish then upgrade then it fails.

Sine the DBs are separated the migration fixes have to be explicitly told which DB db:migrate:up/down:main or db:migrate:up/down:ci etc.

database main = gitlabgitlabhq_production and ci = gitlabhq_production_ci

20230130175512 seems to be sent_notifications and main DB only.
20230130104819 seems to be ci_build_needs and is on both DBs main and ci?

Restore schema

sudo gitlab-psql -c “DELETE FROM schema_migrations WHERE version IN (‘20230130175512’, ‘20230130104819’)”
sudo gitlab-rails db:migrate:up VERSION=20230130175512 *this works
sudo gitlab-rails db:migrate:up VERSION=20230130104819 *this will migrate main and has lock connection for ci.

Re-schedule background migrations

sudo gitlab-rails db:migrate:down VERSION=20230130202201 works for main only and successful
sudo gitlab-rails db:migrate:down VERSION=20230130110855 only works for ci? main says NO.
sudo gitlab-rails db:migrate:up VERSION=20230130202201 successful.
sudo gitlab-rails db:migrate:up VERSION=20230130110855 column doesn’t exist and fails

In order to get the column created in ci:
sudo gitlab-psql -c “DELETE FROM schema_migrations WHERE version = ‘20230130104819’;”
vim /opt/gitlab/embedded/service/gitlab-rails/db/migrate/20230130104819_initialize_conversion_of_ci_build_needs_to_bigint.rb
comment out the 3rd stanza for down.
gitlab-rake db:migrate:down:ci VERSION=20230130104819
uncomment the the file above
gitlab-rake db:migrate:up:ci VERSION=20230130104819

then reschedule the migration:
sudo gitlab-rails db:migrate:down:ci VERSION=20230130110855
sudo gitlab-rails db:migrate:up:ci VERSION=20230130110855

this seems to work and can log in to see the migrations finish. However during an upgrade it fails with:
PG::UniqueViolation: ERROR: could not create unique index “index_ci_build_needs_on_id_convert_to_bigint”
DETAIL: Key (id_convert_to_bigint)=(0) is duplicated.
This appears to be on the main db.
I’ve referenced this page.
Have not been able to get passed this issue and have had to roll back to a snapshot of 16.1.5(thank goodness). Would anyone have any thoughts, ideas, suggestions.

Thanks,
Wayne

2

It’s been a bit. So for anyone out there that have split their Database be sure to run the “truncate legacy tables scripts” or the migration fix will not work and upgrades will fail.
gitlab:db:truncate_legacy_tables:main will truncate the CI tables in Main database.
gitlab:db:truncate_legacy_tables:ci will truncate the Main tables in CI database.

After these tables are truncated then do the migration fix and proceed to upgrade as normal. Everything is working as expected now.

Thanks for reading :slight_smile:
Wayne