Ok, fixed it. The migration was failing with
PG::UndefinedObject: ERROR: index "index_ci_sources_pipelines_on_source_job_id" does not exist
during gitlab-ctl reconfigure
I ran the migrations with verbose tracing:
gitlab-rake db:migrate --trace
and found a bit more detail:
-- execute("DROP INDEX index_ci_sources_pipelines_on_source_job_id")
rake aborted!
StandardError: An error has occurred, all later migrations canceled:
PG::UndefinedObject: ERROR: index "index_ci_sources_pipelines_on_source_job_id" does not exist
The logs also pointed me to the migration file
/opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20210713042153_finalize_ci_sources_pipelines_bigint_conversion.rb
Looking in that file we see the the migration is failing because it cant drop an index that doesn’t exist.
I commented out the line in migration file and re-ran the migration. Problem solved except that it also wanted to remove a foreign key that doesn’t exist. So comment out that line as well.
Now all the migrations complete successfully.
gitlab-rake db:migrate:status confirms all migrations are up, and gitlab health is happy again after a reconfigure.