Upgrade to 16.5.1 migrations failed

How to Use GitLab Self-managed
1

Hi,

I’ve applied the latest 16.5.1-ee update to a self-hosted Gitlab running on Centos 7. The update failed with the key error message appearing to be:

main: -- execute("ALTER INDEX index_ci_sources_pipelines_on_pipeline_id RENAME TO temp_index_ci_sources_pipelines_on_pipeline_id")
rake aborted!
StandardError: An error has occurred, all later migrations canceled:

PG::UndefinedTable: ERROR:  relation "index_ci_sources_pipelines_on_pipeline_id" does not exist

I’ve had something similar happen before, Database migration failed during update to 14.2.2 , but the fix this time is not quite so apparent. I’m digging in to see if I can manually create the missing index, but I wondered if anyone else has encountered this?

It is a bit concerning that my table structure looks different to what others have based on: DB migration fails after Omnibus upgrade to 14.2.2 with PG::UndefinedObject Error (#339844) · Issues · GitLab.org / GitLab · GitLab

Mine looks like:

gitlabhq_production=> \d+ ci_sources_pipelines;
                                                               Table "public.ci_sources_pipelines"
                Column                |  Type   | Collation | Nullable |                     Default                      | Storage | Stats target | Description 
--------------------------------------+---------+-----------+----------+--------------------------------------------------+---------+--------------+-------------
 id                                   | integer |           | not null | nextval('ci_sources_pipelines_id_seq'::regclass) | plain   |              | 
 project_id                           | integer |           |          |                                                  | plain   |              | 
 pipeline_id                          | integer |           |          |                                                  | plain   |              | 
 source_project_id                    | integer |           |          |                                                  | plain   |              | 
 source_pipeline_id                   | integer |           |          |                                                  | plain   |              | 
 source_job_id                        | bigint  |           |          |                                                  | plain   |              | 
 partition_id                         | bigint  |           | not null |                                                  | plain   |              | 
 source_partition_id                  | bigint  |           | not null |                                                  | plain   |              | 
 pipeline_id_convert_to_bigint        | bigint  |           |          |                                                  | plain   |              | 
 source_pipeline_id_convert_to_bigint | bigint  |           |          |                                                  | plain   |              | 
Indexes:
    "ci_sources_pipelines_pkey" PRIMARY KEY, btree (id)
    "index_ci_pipeline_source_pipelines_on_pipeline_id" btree (pipeline_id)
    "index_ci_pipeline_source_pipelines_on_project_id" btree (project_id)
    "index_ci_pipeline_source_pipelines_on_source_pipeline_id" btree (source_pipeline_id)
    "index_ci_pipeline_source_pipelines_on_source_project_id" btree (source_project_id)
    "index_ci_sources_pipelines_on_pipeline_id_bigint" btree (pipeline_id_convert_to_bigint)
    "index_ci_sources_pipelines_on_source_job_id" btree (source_job_id)
    "index_ci_sources_pipelines_on_source_partition_id_source_job_id" btree (source_partition_id, source_job_id)
    "index_ci_sources_pipelines_on_source_pipeline_id_bigint" btree (source_pipeline_id_convert_to_bigint)
Foreign-key constraints:
    "fk_1df371767f" FOREIGN KEY (source_pipeline_id_convert_to_bigint) REFERENCES ci_pipelines(id) ON DELETE CASCADE
    "fk_be5624bf37_p" FOREIGN KEY (source_partition_id, source_job_id) REFERENCES p_ci_builds(partition_id, id) ON UPDATE CASCADE ON DELETE CASCADE
    "fk_c1b5dc6b6f" FOREIGN KEY (pipeline_id_convert_to_bigint) REFERENCES ci_pipelines(id) ON DELETE CASCADE
    "fk_d4e29af7d7" FOREIGN KEY (source_pipeline_id) REFERENCES ci_pipelines(id) ON DELETE CASCADE
    "fk_e1bad85861" FOREIGN KEY (pipeline_id) REFERENCES ci_pipelines(id) ON DELETE CASCADE
Triggers:
    trigger_68d7b6653c7d BEFORE INSERT OR UPDATE ON ci_sources_pipelines FOR EACH ROW EXECUTE FUNCTION trigger_68d7b6653c7d()
Access method: heap
2

So I executed gitlab-rake db:migrate --trace to see where the migrations were failing. This lead me to the migrations file /opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20231004091113_swap_columns_for_ci_sources_pipelines_pipeline_id_bigint.rb where I could see that two indexes to be migrated did not exist in the db.

I thought about it for a while and after checking that the other tables and indexes and foreign keys existed I decided to manually create the indexes.

> gitlab-rails db
CREATE INDEX index_ci_sources_pipelines_on_pipeline_id ON ci_sources_pipelines USING btree (pipeline_id);
CREATE INDEX index_ci_sources_pipelines_on_source_pipeline_id ON ci_sources_pipelines USING btree (source_pipeline_id);

Then I ran the migration with gitlab-rake db:migrate --trace again and it succeeded. Gitlab Admin->Monitoring->Health reports the installation is healthy.

Now, is there any way to check that the actual schema matches what it should be?
Thanks for reading!

2 Likes