Gitlab-ce 17.3.7-ce.0 upgrade to 17.5.5-ce.0 failed, insert or update on table “personal_access_tokens” violates foreign key constraint “fk_da676c7ca5”

Problem to solve

Gitlab-ce 17.3.7-ce.0 upgrade to 17.5.5-ce.0 failed

DB migrate log:

rake aborted!
StandardError: An error has occurred, all later migrations canceled:

PG::ForeignKeyViolation: ERROR:  insert or update on table "personal_access_tokens" violates foreign key constraint "fk_da676c7ca5"
DETAIL:  Key (organization_id)=(1) is not present in table "organizations".
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:306:in `block in add_concurrent_foreign_key'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/timeout_helpers.rb:31:in `disable_statement_timeout'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:305:in `add_concurrent_foreign_key'
/opt/gitlab/embedded/service/gitlab-rails/db/migrate/20240805080941_add_organization_id_to_personal_access_tokens.rb:16:in `up'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb:33:in `block in exec_migration'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/query_analyzer.rb:83:in `within'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb:30:in `exec_migration'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/automatic_lock_writes_on_tables.rb:21:in `exec_migration'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/lock_retry_mixin.rb:54:in `ddl_transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/runner_backoff/active_record_mixin.rb:21:in `execute_migration_in_transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/pg_backend_pid.rb:28:in `block in with_advisory_lock_connection'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/pg_backend_pid.rb:25:in `with_advisory_lock_connection'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:138:in `configure_database'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:107:in `configure_pg_databases'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:94:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:25:in `load'
/opt/gitlab/embedded/bin/bundle:25:in `<main>'

Caused by:
ActiveRecord::InvalidForeignKey: PG::ForeignKeyViolation: ERROR:  insert or update on table "personal_access_tokens" violates foreign key constraint "fk_da676c7ca5"
DETAIL:  Key (organization_id)=(1) is not present in table "organizations".
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:306:in `block in add_concurrent_foreign_key'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/timeout_helpers.rb:31:in `disable_statement_timeout'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:305:in `add_concurrent_foreign_key'
/opt/gitlab/embedded/service/gitlab-rails/db/migrate/20240805080941_add_organization_id_to_personal_access_tokens.rb:16:in `up'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb:33:in `block in exec_migration'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/query_analyzer.rb:83:in `within'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb:30:in `exec_migration'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/automatic_lock_writes_on_tables.rb:21:in `exec_migration'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/lock_retry_mixin.rb:54:in `ddl_transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/runner_backoff/active_record_mixin.rb:21:in `execute_migration_in_transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/pg_backend_pid.rb:28:in `block in with_advisory_lock_connection'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/pg_backend_pid.rb:25:in `with_advisory_lock_connection'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:138:in `configure_database'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:107:in `configure_pg_databases'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:94:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:25:in `load'
/opt/gitlab/embedded/bin/bundle:25:in `<main>'

Caused by:
PG::ForeignKeyViolation: ERROR:  insert or update on table "personal_access_tokens" violates foreign key constraint "fk_da676c7ca5"
DETAIL:  Key (organization_id)=(1) is not present in table "organizations".
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:306:in `block in add_concurrent_foreign_key'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/timeout_helpers.rb:31:in `disable_statement_timeout'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:305:in `add_concurrent_foreign_key'
/opt/gitlab/embedded/service/gitlab-rails/db/migrate/20240805080941_add_organization_id_to_personal_access_tokens.rb:16:in `up'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb:33:in `block in exec_migration'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/query_analyzer.rb:83:in `within'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb:30:in `exec_migration'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/automatic_lock_writes_on_tables.rb:21:in `exec_migration'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/lock_retry_mixin.rb:54:in `ddl_transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/runner_backoff/active_record_mixin.rb:21:in `execute_migration_in_transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/pg_backend_pid.rb:28:in `block in with_advisory_lock_connection'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/pg_backend_pid.rb:25:in `with_advisory_lock_connection'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:138:in `configure_database'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:107:in `configure_pg_databases'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:94:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:25:in `load'
/opt/gitlab/embedded/bin/bundle:25:in `<main>'
Tasks: TOP => db:migrate
(See full trace by running task with --trace)
Running db:migrate rake task
main: == [advisory_lock_connection] object_id: 49560, pg_backend_pid: 579
main: == 20240805080941 AddOrganizationIdToPersonalAccessTokens: migrating ==========
main: -- transaction_open?(nil)
main:    -> 0.0000s
main: -- add_column(:personal_access_tokens, :organization_id, :bigint, {:default=>1, :null=>false, :if_not_exists=>true})
main:    -> 0.0653s
main: -- transaction_open?(nil)
main:    -> 0.0000s
main: -- view_exists?(:postgres_partitions)
main:    -> 0.0009s
main: -- index_exists?(:personal_access_tokens, :organization_id, {:name=>"index_personal_access_tokens_on_organization_id", :algorithm=>:concurrently})
main:    -> 0.0309s
main: -- Index not created because it already exists (this may be due to an aborted migration or similar): table_name: personal_access_tokens, column_name: organization_id
main: -- transaction_open?(nil)
main:    -> 0.0000s
main: -- execute("SET statement_timeout TO 0")
main:    -> 0.0004s
main: -- execute("ALTER TABLE personal_access_tokens VALIDATE CONSTRAINT fk_da676c7ca5;")
main: -- execute("RESET statement_timeout")
main:    -> 0.0006s
main: == [advisory_lock_connection] object_id: 49560, pg_backend_pid: 579

gitlab-rails console output:

irb(main):038:0> group1 = Group.find_by_id(5)
=> #<Group id:5 @goptorg>
irb(main):039:0> group1.organization_id
=> 1

For the same reason insert or update on table “personal_access_tokens” violates foreign key constraint “fk_da676c7ca5”, I cannot create any access token

Versions

Please add an x whether options apply, and add the version information.

  • Self-managed
  • GitLab.com SaaS
  • Dedicated

Versions

System information
System:
Current User:   git
Using RVM:      no
Ruby Version:   3.1.5p253
Gem Version:    3.5.11
Bundler Version:2.5.11
Rake Version:   13.0.6
Redis Version:  7.0.15
Sidekiq Version:7.1.6
Go Version:     unknown

GitLab information
Version:        17.3.7
Revision:       4880aeec201
Directory:      /opt/gitlab/embedded/service/gitlab-rails
DB Adapter:     PostgreSQL
DB Version:     14.11
URL:            http://192.168.1.100:8081
HTTP Clone URL: http://192.168.1.100:8081/some-group/some-project.git
SSH Clone URL:  ssh://git@192.168.1.100:8083/some-group/some-project.git
Using LDAP:     no
Using Omniauth: yes
Omniauth Providers: 

GitLab Shell
Version:        14.38.0
Repository storages:
- default:      unix:/var/opt/gitlab/gitaly/gitaly.socket
GitLab Shell path:              /opt/gitlab/embedded/service/gitlab-shell

Gitaly
- default Address:      unix:/var/opt/gitlab/gitaly/gitaly.socket
- default Version:      17.3.7
- default Git Version:  2.45.2