One of the database migrations failed during my update of a CentOS 7, self managed, omnibus installation from version 14.10.3 to 15.0.0. My server appears to be working after a restart, but here’s what I get when I try sudo
1 Like
Hit return prematurely in prior post…
sudo gitlab-rake db:migrate
== 20220412135446 AddUniqueFingerprintSha256IndexToKey: migrating =============
-- transaction_open?()
-> 0.0000s
-- index_exists?(:keys, :fingerprint_sha256, {:unique=>true, :name=>"index_keys_on_fingerprint_sha256_unique", :algorithm=>:concurrently})
-> 0.0055s
-- Index being recreated because the existing version was INVALID: table_name: keys, column_name: fingerprint_sha256
-- transaction_open?()
-> 0.0000s
-- indexes(:keys)
-> 0.0037s
-- execute("SET statement_timeout TO 0")
-> 0.0002s
-- remove_index(:keys, {:algorithm=>:concurrently, :name=>"index_keys_on_fingerprint_sha256_unique"})
-> 0.1612s
-- execute("RESET statement_timeout")
-> 0.0003s
-- execute("SET statement_timeout TO 0")
-> 0.0001s
-- add_index(:keys, :fingerprint_sha256, {:unique=>true, :name=>"index_keys_on_fingerprint_sha256_unique", :algorithm=>:concurrently})
-- execute("RESET statement_timeout")
-> 0.0003s
rake aborted!
StandardError: An error has occurred, all later migrations canceled:
PG::UniqueViolation: ERROR: could not create unique index "index_keys_on_fingerprint_sha256_unique"
DETAIL: Key (fingerprint_sha256)=(****) is duplicated.
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:167:in `block in add_concurrent_index'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:388:in `disable_statement_timeout'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:166:in `add_concurrent_index'
/opt/gitlab/embedded/service/gitlab-rails/db/migrate/20220412135446_add_unique_fingerprint_sha256_index_to_key.rb:10:in `up'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/lock_retry_mixin.rb:36:in `ddl_transaction'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Caused by:
ActiveRecord::RecordNotUnique: PG::UniqueViolation: ERROR: could not create unique index "index_keys_on_fingerprint_sha256_unique"
DETAIL: Key (fingerprint_sha256)=(****) is duplicated.
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:167:in `block in add_concurrent_index'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:388:in `disable_statement_timeout'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:166:in `add_concurrent_index'
/opt/gitlab/embedded/service/gitlab-rails/db/migrate/20220412135446_add_unique_fingerprint_sha256_index_to_key.rb:10:in `up'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/lock_retry_mixin.rb:36:in `ddl_transaction'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Caused by:
PG::UniqueViolation: ERROR: could not create unique index "index_keys_on_fingerprint_sha256_unique"
DETAIL: Key (fingerprint_sha256)=(****) is duplicated.
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:167:in `block in add_concurrent_index'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:388:in `disable_statement_timeout'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:166:in `add_concurrent_index'
/opt/gitlab/embedded/service/gitlab-rails/db/migrate/20220412135446_add_unique_fingerprint_sha256_index_to_key.rb:10:in `up'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/lock_retry_mixin.rb:36:in `ddl_transaction'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Tasks: TOP => db:migrate
(See full trace by running task with --trace)
This problem is also preventing successive migrations:
sudo -u git -H gitlab-rake db:migrate:status RAILS_ENV=production
...
up 20220412060931 Add nullify build data trigger on merge request metrics
down 20220412135446 Add unique fingerprint sha256 index to key
down 20220412140446 Add unique fingerprint sha256 index to group deploy key
down 20220412140755 Drop unique fingerprint md5 index from key
down 20220412141020 Drop unique fingerprint md5 index from group deploy key
up 20220412143551 Add partial index on unencrypted integrations
up 20220412143552 Consume remaining encrypt integration property jobs
up 20220412171810 Add otp secret expires at
up 20220413011328 Remove partial index on unencrypted integrations
down 20220413054910 Backfill delayed group deletion
up 20220413075921 Update index on packages build infos
down 20220413124200 Add view for per table autovacuum status
down 20220413164146 Remove max seats used indices
down 20220413235818 Add max seats used changed at to gitlab subscriptions
down 20220414203622 Add index for columns user custom attribute
down 20220415015143 Replace iterations cadence date range constraint
down 20220415124802 Remove job artifact deprecated geo fields
down 20220415124804 Remove job artifact deleted event table
down 20220419223906 Add arkose namespace to application settings
down 20220420034519 Add text limit to in product marketing email campaign
down 20220420061439 Add notes null discussion id temp index
down 20220420061450 Backfill null note discussion ids
down 20220420135245 Fix batched background migration default arguments
down 20220420135946 Update batched background migration arguments
down 20220420173247 Add group inheritance type to pe authorizable
down 20220420192542 Add id for cleanup index packages package files
down 20220420214703 Schedule backfill draft status on merge requests corrected regex
down 20220421114418 Make fingerprint nullable for group deploy key
down 20220421141342 Add allowed plans to ci runners
down 20220421144758 Change dotenv plan limits for old plans
down 20220421180321 Add last cleanup deleted tags count to container repository
down 20220422121443 Add async index for group activity events
down 20220422200633 Fix view for per table autovacuum status
down 20220422220507 Remove tmp index supporting leaky regex cleanup
down 20220425111114 Add async index for project activity events
down 20220425111453 Add async index to events on group id and
down 20220425120604 Create packages cleanup policies
down 20220425121410 Add temporary index for backfill integrations enable ssl verification
down 20220425121435 Backfill integrations enable ssl verification
down 20220426130217 Add max export size to application settings
down 20220426185933 Backfill deployments finished at
down 20220428133724 Schedule expire o auth tokens
down 20220502015011 Clean up fix merge request diff commit users
down 20220502125053 Recreate index for project group link with group id and project
down 20220502150408 Add slack integrations bot columns
down 20220502152633 Add slack integrations bot user id text limit
down 20220502173045 Reset too many tags skipped registry imports
down 20220503073401 Recreate index for group group link with both group ids
down 20220503102855 Add namespace ci cd settings table
down 20220503114353 Prepare confidential note index
down 20220503134610 Remove requirements management test reports requirement
down 20220504042847 Add slack signing key to application settings
down 20220504083348 Add indexes for primary email post regression cleanup migration
down 20220504083836 Cleanup after fixing regression with new users emails
down 20220504084136 Drop temporary indexes for primary email post regression cleanup
down 20220504140036 Add container registry size to namespace root storage statistics
down 20220505022001 Add index to deployments on created at cluster id and project
down 20220505044348 Fix automatic iterations cadences start date
down 20220505053504 Remove namespaces id parent id inverse partial index
down 20220505060011 Remove namespaces id parent id partial index
down 20220505092254 Add allow stale runner pruning index to namespace ci cd settings
down 20220505174658 Update index on alerts to exclude null fingerprints
down 20220505193512 Add not null constraint to issues work item type
down 20220506180411 Add index to vulnerability feedback finding uuid
down 20220506201639 Remove devops adoption security scan succeeded column
up 20220507204024 Add separated caches option to project ci settings
down 20220510003916 Remove tmp empty traversal ids root namespace index
down 20220510004501 Remove tmp empty traversal ids child namespace index
down 20220511090324 Add last consistency check cursors to vsa aggregations
down 20220511151646 Add exclude from free user cap to namespace settings
down 20220511152439 Remove not null constraint from work item type
down 20220511191502 Add registry migration guard thresholds to application settings
down 20220511191503 Add pipeline create limit per project user sha to application settings
down 20220513043344 Reschedule expire o auth tokens
down 20220517144749 Remove vulnerability approval rules
Any help would be greatly appreciated.
I resolved this by accessing the database using
sudo gitlab-rails dbconsole --database main
Removing the fingerprint_sha256 columns from the keys and group_deploy_keys tables
ALTER TABLE keys DROP COLUMN fingerprint_sha256;
ALTER TABLE group_deploy_keys DROP COLUMN fingerprint_sha256;
Adding the columns back using
ALTER TABLE keys ADD COLUMN fingerprint_sha256 bytea;
ALTER TABLE group_deploy_keys ADD COLUMN fingerprint_sha256 bytea;
Ran the migrations
sudo gitlab-rake db:migrate
Reconfigured GitLab
sudo gitlab-ctl reconfigure
Then all was well and I was able to update.
1 Like