Hi, Updated my omnibus installation to 14.2.2 today but a database migration failed with the following error ( just the key lines):
rails_migration[gitlab-rails] (gitlab::database_migrations line 51) had an error: Mixlib::ShellOut::ShellCommandFailed: bash[migrate gitlab-rails database] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/resources/rails_migration.rb line 16) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
---- Begin output of "bash" "/tmp/chef-script20210901-17902-1vvmj2s" ----
STDOUT: rake aborted!
StandardError: An error has occurred, all later migrations canceled:
PG::UndefinedObject: ERROR: index "index_ci_sources_pipelines_on_source_job_id" does not exist
Looking in the database at the ci_sources_pipelines table I see that the index is not there but there is an index called index_ci_sources_pipelines_on_source_job_id_convert_to_bigint.
Ok, fixed it. The migration was failing with PG::UndefinedObject: ERROR: index "index_ci_sources_pipelines_on_source_job_id" does not exist
during gitlab-ctl reconfigure
I ran the migrations with verbose tracing: gitlab-rake db:migrate --trace
and found a bit more detail:
-- execute("DROP INDEX index_ci_sources_pipelines_on_source_job_id")
rake aborted!
StandardError: An error has occurred, all later migrations canceled:
PG::UndefinedObject: ERROR: index "index_ci_sources_pipelines_on_source_job_id" does not exist
The logs also pointed me to the migration file /opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20210713042153_finalize_ci_sources_pipelines_bigint_conversion.rb
Looking in that file we see the the migration is failing because it cant drop an index that doesn’t exist.
I commented out the line in migration file and re-ran the migration. Problem solved except that it also wanted to remove a foreign key that doesn’t exist. So comment out that line as well.
Now all the migrations complete successfully.
gitlab-rake db:migrate:status confirms all migrations are up, and gitlab health is happy again after a reconfigure.
Thanks so much @dave.robertson for posting this - and the fix - you just made my day (only google result for this)
I’ve logged a bug for it on the GitLab issue tracker!
I fixed it by adding the wanted index and constraint. For some reason, they were not present on the db.
go with
gitlab-rails db
ALTER TABLE ONLY ci_sources_pipelines
ADD CONSTRAINT fk_be5624bf37 FOREIGN KEY (source_job_id) REFERENCES ci_builds(id) ON DELETE CASCADE;
CREATE INDEX index_ci_sources_pipelines_on_source_job_id ON ci_sources_pipelines USING btree (source_job_id);
Big trouble with upgrading my version, I have a problem related to the same step, but different cause.
Please help:
end
System Info:
------------
chef_version=15.17.4
platform=ubuntu
platform_version=18.04
ruby=ruby 2.7.4p191 (2021-07-07 revision a21a3b7d23) [x86_64-linux]
program_name=/opt/gitlab/embedded/bin/chef-client
executable=/opt/gitlab/embedded/bin/chef-client
Running handlers:
There was an error running gitlab-ctl reconfigure:
rails_migration[gitlab-rails] (gitlab::database_migrations line 51) had an error: Mixlib::ShellOut::ShellCommandFailed: bash[migrate gitlab-rails database] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/resources/rails_migration.rb line 16) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
---- Begin output of "bash" "/tmp/chef-script20211108-5587-y7o44i" ----
STDOUT: rake aborted!
ActiveRecord::ConnectionNotEstablished: could not connect to server: No such file or directory
Is the server running locally and accepting
connections on Unix domain socket "/var/opt/gitlab/postgresql/.s.PGSQL.5432"?
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:106:in `block in read_write'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:167:in `retry_with_backoff'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:105:in `read_write'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:118:in `write_using_load_balancer'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:88:in `method_missing'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:60:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Interesting. I was running the exact same version also on Centos 7. Unfortunately this wasn’t the final solution. I had further errors related to table constrains.
I ended up rolling it back, because I couldn’t afford keeping it offline for too long.
I’m still looking for a definitive solution.
The only difference was that I was upgrading to 14.5
As reference:
I haven’t yet tried the solution offered by BennyK.
And I’m not sure I’m actually going to try it anyhow before getting an official solution from the gitlab developers. I can’t estimate the risk and I can’t know the consequences - i.e. if I have to make further manual changes on the next upgrade and so on.
I have no idea how poking around in your suggested steps fixed my system. My errors were slightly different and much less verbose/helpful. However, just poking my system with the gitlab-rake db:migrate steps seems to have fixed my issue. You guys are lifesavers.
My issue started when I noticed that my gitlab installation wasn’t updating anymore, so I ran the following to fix a certificate issue, then all hell broke loose which lead me here:
curl -L https://packages.gitlab.com/gitlab/gitlab-ee/gpgkey | apt-key add -
apt-get update
apt-get upgrade
echo "Sorry sucka - now your system will be annoyingly broken for a week until you find this forum post"