Database migration failed during update to 14.2.2

Hi, Updated my omnibus installation to 14.2.2 today but a database migration failed with the following error ( just the key lines):

rails_migration[gitlab-rails] (gitlab::database_migrations line 51) had an error: Mixlib::ShellOut::ShellCommandFailed: bash[migrate gitlab-rails database] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/resources/rails_migration.rb line 16) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
---- Begin output of "bash"  "/tmp/chef-script20210901-17902-1vvmj2s" ----
STDOUT: rake aborted!
StandardError: An error has occurred, all later migrations canceled:

PG::UndefinedObject: ERROR:  index "index_ci_sources_pipelines_on_source_job_id" does not exist

Looking in the database at the ci_sources_pipelines table I see that the index is not there but there is an index called index_ci_sources_pipelines_on_source_job_id_convert_to_bigint.

gitlabhq_production-# \d ci_sources_pipelines
                                         Table "public.ci_sources_pipelines"
             Column              |  Type   | Collation | Nullable |                     Default                      
 id                              | integer |           | not null | nextval('ci_sources_pipelines_id_seq'::regclass)
 project_id                      | integer |           |          | 
 pipeline_id                     | integer |           |          | 
 source_project_id               | integer |           |          | 
 source_job_id                   | integer |           |          | 
 source_pipeline_id              | integer |           |          | 
 source_job_id_convert_to_bigint | bigint  |           |          | 
    "ci_sources_pipelines_pkey" PRIMARY KEY, btree (id)
    "index_ci_pipeline_source_pipelines_on_pipeline_id" btree (pipeline_id)
    "index_ci_pipeline_source_pipelines_on_project_id" btree (project_id)
    "index_ci_pipeline_source_pipelines_on_source_job_id" btree (source_job_id)
    "index_ci_pipeline_source_pipelines_on_source_pipeline_id" btree (source_pipeline_id)
    "index_ci_pipeline_source_pipelines_on_source_project_id" btree (source_project_id)
    "index_ci_sources_pipelines_on_source_job_id_convert_to_bigint" btree (source_job_id_convert_to_bigint)

I have brought the gitlab instance up using gitlab-ctl start, but the health check is unhappy since migrations are still pending.

Whats my next move? Roll back and try the upgrade again?

Thanks for any advice.

Ok, fixed it. The migration was failing with
PG::UndefinedObject: ERROR: index "index_ci_sources_pipelines_on_source_job_id" does not exist
during gitlab-ctl reconfigure

I ran the migrations with verbose tracing:
gitlab-rake db:migrate --trace
and found a bit more detail:

-- execute("DROP INDEX index_ci_sources_pipelines_on_source_job_id")
rake aborted!
StandardError: An error has occurred, all later migrations canceled:
PG::UndefinedObject: ERROR: index "index_ci_sources_pipelines_on_source_job_id" does not exist

The logs also pointed me to the migration file

Looking in that file we see the the migration is failing because it cant drop an index that doesn’t exist.
I commented out the line in migration file and re-ran the migration. Problem solved except that it also wanted to remove a foreign key that doesn’t exist. So comment out that line as well.

Now all the migrations complete successfully.

gitlab-rake db:migrate:status confirms all migrations are up, and gitlab health is happy again after a reconfigure.


Thanks so much @dave.robertson for posting this - and the fix - you just made my day (only google result for this)
I’ve logged a bug for it on the GitLab issue tracker!

@dave.robertson out of interest, what version were you running before the update?

14.1.2-ee on Centos 7

I am having the same problem. Trying to upgrade from 14.1.1 to 14.2.3-ce.0 in docker.

I fixed it by adding the wanted index and constraint. For some reason, they were not present on the db.

go with

gitlab-rails db

ALTER TABLE ONLY ci_sources_pipelines
    ADD CONSTRAINT fk_be5624bf37 FOREIGN KEY (source_job_id) REFERENCES ci_builds(id) ON DELETE CASCADE;
CREATE INDEX index_ci_sources_pipelines_on_source_job_id ON ci_sources_pipelines USING btree (source_job_id);