Failed to reconfigure when updating from 17.2 to 17.2.1

How to Use GitLab
1

I’m trying to update to 17.2.1 from 17.2.0, but I ran into some problems when updating.
The issue seems to be related to migrating the database.

os : ubuntu 24.04
gitlab 17.2.0 → 17.2.1

see log:

[2024-07-28T09:06:52+00:00] INFO: Running queued delayed notifications before re-raising exception
[2024-07-28T09:06:52+00:00] ERROR: Running exception handlers
[2024-07-28T09:06:52+00:00] ERROR: Exception handlers complete
[2024-07-28T09:06:52+00:00] FATAL: Stacktrace dumped to /opt/gitlab/embedded/cookbooks/cache/cinc-stacktrace.out
[2024-07-28T09:06:52+00:00] FATAL: ---------------------------------------------------------------------------------------
[2024-07-28T09:06:52+00:00] FATAL: PLEASE PROVIDE THE CONTENTS OF THE stacktrace.out FILE (above) IF YOU FILE A BUG REPORT
[2024-07-28T09:06:52+00:00] FATAL: ---------------------------------------------------------------------------------------
[2024-07-28T09:06:52+00:00] FATAL: Mixlib::ShellOut::ShellCommandFailed: rails_migration[gitlab-rails] (gitlab::database_migrations line 51) had an error: Mixlib::ShellOut::ShellCommandFailed: bash_hide_env[migrate gitlab-rails database] (gitlab::database_migrations line 20) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
---- Begin output of "bash"  ----
STDOUT: rake aborted!
StandardError: An error has occurred, all later migrations canceled:

key not found: "project_repository_states_id_seq"
Did you mean?  "project_repository_storage_moves_id_seq"
               "project_repositories_id_seq"
/opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20240709014000_fix_sequence_owners_for_ci_builds.rb:668:in `fetch'
/opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20240709014000_fix_sequence_owners_for_ci_builds.rb:668:in `block in up'
/opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20240709014000_fix_sequence_owners_for_ci_builds.rb:667:in `each'
/opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20240709014000_fix_sequence_owners_for_ci_builds.rb:667:in `up'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb:33:in `block in exec_migration'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/query_analyzer.rb:40:in `within'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb:30:in `exec_migration'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/automatic_lock_writes_on_tables.rb:21:in `exec_migration'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/lock_retry_mixin.rb:54:in `ddl_transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/runner_backoff/active_record_mixin.rb:21:in `execute_migration_in_transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/pg_backend_pid.rb:15:in `block in with_advisory_lock_connection'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/pg_backend_pid.rb:12:in `with_advisory_lock_connection'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:136:in `configure_database'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:107:in `configure_pg_databases'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:94:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:25:in `load'
/opt/gitlab/embedded/bin/bundle:25:in `<main>'

Caused by:
KeyError: key not found: "project_repository_states_id_seq"
Did you mean?  "project_repository_storage_moves_id_seq"
               "project_repositories_id_seq"
/opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20240709014000_fix_sequence_owners_for_ci_builds.rb:668:in `fetch'
/opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20240709014000_fix_sequence_owners_for_ci_builds.rb:668:in `block in up'
/opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20240709014000_fix_sequence_owners_for_ci_builds.rb:667:in `each'
/opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20240709014000_fix_sequence_owners_for_ci_builds.rb:667:in `up'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb:33:in `block in exec_migration'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/query_analyzer.rb:40:in `within'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb:30:in `exec_migration'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/automatic_lock_writes_on_tables.rb:21:in `exec_migration'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/lock_retry_mixin.rb:54:in `ddl_transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/runner_backoff/active_record_mixin.rb:21:in `execute_migration_in_transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/pg_backend_pid.rb:15:in `block in with_advisory_lock_connection'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/pg_backend_pid.rb:12:in `with_advisory_lock_connection'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:136:in `configure_database'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:107:in `configure_pg_databases'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:94:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:25:in `load'
/opt/gitlab/embedded/bin/bundle:25:in `<main>'
Tasks: TOP => db:migrate
(See full trace by running task with --trace)
Running db:migrate rake task
main: == [advisory_lock_connection] object_id: 57080, pg_backend_pid: 307291
main: == 20240709014000 FixSequenceOwnersForCiBuilds: migrating =====================
main: -- select_all("      SELECT s.relname AS seq_name,\n             t.relname AS table_name,\n             a.attname AS column_name\n      FROM pg_class s\n      JOIN pg_depend d ON d.objid = s.oid\n      AND d.classid = 'pg_class'::regclass\n      AND d.refclassid = 'pg_class'::regclass\n      JOIN pg_catalog.pg_class t ON t.oid = d.refobjid\n      JOIN pg_catalog.pg_namespace n ON n.oid = t.relnamespace\n      JOIN pg_catalog.pg_attribute a ON a.attrelid = t.oid\n      AND a.attnum = d.refobjsubid\n      WHERE s.relkind = 'S'\n        AND (t.relname = 'ci_builds'\n             OR t.relname = 'p_ci_builds');\n")
main:    -> 0.0782s
main: == [advisory_lock_connection] object_id: 57080, pg_backend_pid: 307291
STDERR: 
---- End output of "bash"  ----
Ran "bash"  returned 1
2

I’ve encountered this as well, upgrading from 17.1.2 to 17.2.1. I’m running on RedHat 8.

I manually ran gitlab-ctl reconfigure I get this in the output:

System Info:
------------
chef_version=18.3.0
platform=redhat
platform_version=8.10
ruby=ruby 3.1.5p253 (2024-04-023 revision 1945f8dc0e) [x86_64-linux]
program_name=/opt/gitlab/embedded/bin/cinc-client
executable=/opt/gitlab/embedded/bin/cinc-client

================================================================================
Error executing action `run` on resource 'rails_migration[gitlab-rails]'
================================================================================

Mixlib::ShellOut::ShellCommandFailed
------------------------------------
bash_hide_env[migrate gitlab-rails database] (gitlab::database_migrations line 20) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
---- Begin output of "bash"  ----
STDOUT: rake aborted!
StandardError: An error has occurred, all later migrations canceled:

key not found: "geo_repository_updated_events_id_seq"
Did you mean?  "geo_node_statuses_id_seq"

I have the whole output of the reconfigure saved as well as the cinc-stacktrace.out file available if looking at that helps.

I stopped and then started gitlab and all seems well.

Bruce

3

wow haha, exactly the same manually gitlab-ctl reconfigure, gave me an error, rebooted the server and everything is normal thanks

4

Update on my situation. I noticed had A LOT of background jobs jobs failing this morning so I decided to revert back to v17.1.2. That worked with out issue after I rebuild my RPM database. Not sure if that contributed my issue or not, but I’m back up and running.