Database migration failed during update to 14.2.2

Hi, Updated my omnibus installation to 14.2.2 today but a database migration failed with the following error ( just the key lines):

rails_migration[gitlab-rails] (gitlab::database_migrations line 51) had an error: Mixlib::ShellOut::ShellCommandFailed: bash[migrate gitlab-rails database] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/resources/rails_migration.rb line 16) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
---- Begin output of "bash"  "/tmp/chef-script20210901-17902-1vvmj2s" ----
STDOUT: rake aborted!
StandardError: An error has occurred, all later migrations canceled:

PG::UndefinedObject: ERROR:  index "index_ci_sources_pipelines_on_source_job_id" does not exist

Looking in the database at the ci_sources_pipelines table I see that the index is not there but there is an index called index_ci_sources_pipelines_on_source_job_id_convert_to_bigint.

gitlabhq_production-# \d ci_sources_pipelines
                                         Table "public.ci_sources_pipelines"
             Column              |  Type   | Collation | Nullable |                     Default                      
---------------------------------+---------+-----------+----------+--------------------------------------------------
 id                              | integer |           | not null | nextval('ci_sources_pipelines_id_seq'::regclass)
 project_id                      | integer |           |          | 
 pipeline_id                     | integer |           |          | 
 source_project_id               | integer |           |          | 
 source_job_id                   | integer |           |          | 
 source_pipeline_id              | integer |           |          | 
 source_job_id_convert_to_bigint | bigint  |           |          | 
Indexes:
    "ci_sources_pipelines_pkey" PRIMARY KEY, btree (id)
    "index_ci_pipeline_source_pipelines_on_pipeline_id" btree (pipeline_id)
    "index_ci_pipeline_source_pipelines_on_project_id" btree (project_id)
    "index_ci_pipeline_source_pipelines_on_source_job_id" btree (source_job_id)
    "index_ci_pipeline_source_pipelines_on_source_pipeline_id" btree (source_pipeline_id)
    "index_ci_pipeline_source_pipelines_on_source_project_id" btree (source_project_id)
    "index_ci_sources_pipelines_on_source_job_id_convert_to_bigint" btree (source_job_id_convert_to_bigint)

I have brought the gitlab instance up using gitlab-ctl start, but the health check is unhappy since migrations are still pending.

Whats my next move? Roll back and try the upgrade again?

Thanks for any advice.
-Dave

Ok, fixed it. The migration was failing with
PG::UndefinedObject: ERROR: index "index_ci_sources_pipelines_on_source_job_id" does not exist
during gitlab-ctl reconfigure

I ran the migrations with verbose tracing:
gitlab-rake db:migrate --trace
and found a bit more detail:

-- execute("DROP INDEX index_ci_sources_pipelines_on_source_job_id")
rake aborted!
StandardError: An error has occurred, all later migrations canceled:
PG::UndefinedObject: ERROR: index "index_ci_sources_pipelines_on_source_job_id" does not exist

The logs also pointed me to the migration file
/opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20210713042153_finalize_ci_sources_pipelines_bigint_conversion.rb

Looking in that file we see the the migration is failing because it cant drop an index that doesn’t exist.
I commented out the line in migration file and re-ran the migration. Problem solved except that it also wanted to remove a foreign key that doesn’t exist. So comment out that line as well.

Now all the migrations complete successfully.

gitlab-rake db:migrate:status confirms all migrations are up, and gitlab health is happy again after a reconfigure.

2 Likes

Thanks so much @dave.robertson for posting this - and the fix - you just made my day (only google result for this)
I’ve logged a bug for it on the GitLab issue tracker!

@dave.robertson out of interest, what version were you running before the update?

14.1.2-ee on Centos 7

I am having the same problem. Trying to upgrade from 14.1.1 to 14.2.3-ce.0 in docker.

I fixed it by adding the wanted index and constraint. For some reason, they were not present on the db.

go with

gitlab-rails db

ALTER TABLE ONLY ci_sources_pipelines
    ADD CONSTRAINT fk_be5624bf37 FOREIGN KEY (source_job_id) REFERENCES ci_builds(id) ON DELETE CASCADE;
CREATE INDEX index_ci_sources_pipelines_on_source_job_id ON ci_sources_pipelines USING btree (source_job_id);