Database migration failed during update to 14.2.2

Hi, Updated my omnibus installation to 14.2.2 today but a database migration failed with the following error ( just the key lines):

rails_migration[gitlab-rails] (gitlab::database_migrations line 51) had an error: Mixlib::ShellOut::ShellCommandFailed: bash[migrate gitlab-rails database] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/resources/rails_migration.rb line 16) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
---- Begin output of "bash"  "/tmp/chef-script20210901-17902-1vvmj2s" ----
STDOUT: rake aborted!
StandardError: An error has occurred, all later migrations canceled:

PG::UndefinedObject: ERROR:  index "index_ci_sources_pipelines_on_source_job_id" does not exist

Looking in the database at the ci_sources_pipelines table I see that the index is not there but there is an index called index_ci_sources_pipelines_on_source_job_id_convert_to_bigint.

gitlabhq_production-# \d ci_sources_pipelines
                                         Table "public.ci_sources_pipelines"
             Column              |  Type   | Collation | Nullable |                     Default                      
---------------------------------+---------+-----------+----------+--------------------------------------------------
 id                              | integer |           | not null | nextval('ci_sources_pipelines_id_seq'::regclass)
 project_id                      | integer |           |          | 
 pipeline_id                     | integer |           |          | 
 source_project_id               | integer |           |          | 
 source_job_id                   | integer |           |          | 
 source_pipeline_id              | integer |           |          | 
 source_job_id_convert_to_bigint | bigint  |           |          | 
Indexes:
    "ci_sources_pipelines_pkey" PRIMARY KEY, btree (id)
    "index_ci_pipeline_source_pipelines_on_pipeline_id" btree (pipeline_id)
    "index_ci_pipeline_source_pipelines_on_project_id" btree (project_id)
    "index_ci_pipeline_source_pipelines_on_source_job_id" btree (source_job_id)
    "index_ci_pipeline_source_pipelines_on_source_pipeline_id" btree (source_pipeline_id)
    "index_ci_pipeline_source_pipelines_on_source_project_id" btree (source_project_id)
    "index_ci_sources_pipelines_on_source_job_id_convert_to_bigint" btree (source_job_id_convert_to_bigint)

I have brought the gitlab instance up using gitlab-ctl start, but the health check is unhappy since migrations are still pending.

Whats my next move? Roll back and try the upgrade again?

Thanks for any advice.
-Dave

Ok, fixed it. The migration was failing with
PG::UndefinedObject: ERROR: index "index_ci_sources_pipelines_on_source_job_id" does not exist
during gitlab-ctl reconfigure

I ran the migrations with verbose tracing:
gitlab-rake db:migrate --trace
and found a bit more detail:

-- execute("DROP INDEX index_ci_sources_pipelines_on_source_job_id")
rake aborted!
StandardError: An error has occurred, all later migrations canceled:
PG::UndefinedObject: ERROR: index "index_ci_sources_pipelines_on_source_job_id" does not exist

The logs also pointed me to the migration file
/opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20210713042153_finalize_ci_sources_pipelines_bigint_conversion.rb

Looking in that file we see the the migration is failing because it cant drop an index that doesn’t exist.
I commented out the line in migration file and re-ran the migration. Problem solved except that it also wanted to remove a foreign key that doesn’t exist. So comment out that line as well.

Now all the migrations complete successfully.

gitlab-rake db:migrate:status confirms all migrations are up, and gitlab health is happy again after a reconfigure.

3 Likes

Thanks so much @dave.robertson for posting this - and the fix - you just made my day (only google result for this)
I’ve logged a bug for it on the GitLab issue tracker!

@dave.robertson out of interest, what version were you running before the update?

14.1.2-ee on Centos 7

I am having the same problem. Trying to upgrade from 14.1.1 to 14.2.3-ce.0 in docker.

I fixed it by adding the wanted index and constraint. For some reason, they were not present on the db.

go with

gitlab-rails db

ALTER TABLE ONLY ci_sources_pipelines
    ADD CONSTRAINT fk_be5624bf37 FOREIGN KEY (source_job_id) REFERENCES ci_builds(id) ON DELETE CASCADE;
CREATE INDEX index_ci_sources_pipelines_on_source_job_id ON ci_sources_pipelines USING btree (source_job_id);

Big trouble with upgrading my version, I have a problem related to the same step, but different cause.
Please help:

  end

    System Info:
    ------------
    chef_version=15.17.4
    platform=ubuntu
    platform_version=18.04
    ruby=ruby 2.7.4p191 (2021-07-07 revision a21a3b7d23) [x86_64-linux]
    program_name=/opt/gitlab/embedded/bin/chef-client
    executable=/opt/gitlab/embedded/bin/chef-client


Running handlers:
There was an error running gitlab-ctl reconfigure:

rails_migration[gitlab-rails] (gitlab::database_migrations line 51) had an error: Mixlib::ShellOut::ShellCommandFailed: bash[migrate gitlab-rails database] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/resources/rails_migration.rb line 16) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
---- Begin output of "bash"  "/tmp/chef-script20211108-5587-y7o44i" ----
STDOUT: rake aborted!
ActiveRecord::ConnectionNotEstablished: could not connect to server: No such file or directory
        Is the server running locally and accepting
        connections on Unix domain socket "/var/opt/gitlab/postgresql/.s.PGSQL.5432"?
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:106:in `block in read_write'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:167:in `retry_with_backoff'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:105:in `read_write'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:118:in `write_using_load_balancer'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:88:in `method_missing'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:60:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'

Interesting. I was running the exact same version also on Centos 7. Unfortunately this wasn’t the final solution. I had further errors related to table constrains.
I ended up rolling it back, because I couldn’t afford keeping it offline for too long.
I’m still looking for a definitive solution.
The only difference was that I was upgrading to 14.5
As reference:

I haven’t yet tried the solution offered by BennyK.

And I’m not sure I’m actually going to try it anyhow before getting an official solution from the gitlab developers. I can’t estimate the risk and I can’t know the consequences - i.e. if I have to make further manual changes on the next upgrade and so on.

Please see my response in DB migration fails after Omnibus upgrade to 14.2.2 with PG::UndefinedObject Error (#339844) · Issues · GitLab.org / GitLab · GitLab. Did the migration fail before for some other reason?

I have no idea how poking around in your suggested steps fixed my system. My errors were slightly different and much less verbose/helpful. However, just poking my system with the gitlab-rake db:migrate steps seems to have fixed my issue. You guys are lifesavers.
My issue started when I noticed that my gitlab installation wasn’t updating anymore, so I ran the following to fix a certificate issue, then all hell broke loose which lead me here:

curl -L https://packages.gitlab.com/gitlab/gitlab-ee/gpgkey | apt-key add -
apt-get update
apt-get upgrade
echo "Sorry sucka - now your system will be annoyingly broken for a week until you find this forum post"