Hello,
Last week, we failed to upgrade Gitlab from version 15.6.1 to 15.6.2.
We can’t restore last version and each reconfigure we do fail. I’ll copy the reconfigure log file at the end of this message.
We can read on this file those lines :
ActiveRecord::StatementInvalid: PG::ObjectNotInPrerequisiteState: ERROR: sequence must have same owner as table it is linked to
/opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20220902204048_move_security_findings_table_to_gitlab_partitions_dynamic_schema.rb:80:in `block in up'
From what we have read here : DB Migration fails on latest update - #2 by jedihomer
and also in other places, we thought it could be the owner of the table security_findings which is different from the sequence. But both have the same :
However, we see at the end of the log file that the reconfigure command try to make
“public.security_findings.id” the owner of the sequence :
execute("ALTER SEQUENCE security_findings_id_seq OWNED BY public.security_findings.id;\n")
Also, in addition to not be able to upgrade our gitlab version, we can’t see our pipeline on the webUI. From the logs, it seems that we are missing the table “ci_pipeline_metadata” :
There is no such table on our database.
We will continue to search for an option this afternoon, and keep this thread up to date. If someone has a clue (or even a solution !), or if you need additional information or file, don’t hesitate.
Thanks and have a good day !
Here is the reconfigure log file :
Logfile created on 2022-12-20 15:37:52 +0100 by logger.rb/v1.4.2
[2022-12-20T15:37:53+01:00] INFO: Started Cinc Zero at chefzero://localhost:1 with repository at /opt/gitlab/embedded (One version per cookbook)
[2022-12-20T15:37:53+01:00] INFO: *** Cinc Client 17.10.0 ***
[2022-12-20T15:37:53+01:00] INFO: Platform: x86_64-linux
[2022-12-20T15:37:53+01:00] INFO: Cinc-client pid: 788723
[2022-12-20T15:37:54+01:00] INFO: Setting the run_list to [“recipe[gitlab]”] from CLI options
[2022-12-20T15:37:54+01:00] INFO: Run List is [recipe[gitlab]]
[2022-12-20T15:37:54+01:00] INFO: Run List expands to [gitlab]
[2022-12-20T15:37:54+01:00] INFO: Starting Cinc Client Run for debian
[2022-12-20T15:37:54+01:00] INFO: Running start handlers
[2022-12-20T15:37:54+01:00] INFO: Start handlers complete.
[2022-12-20T15:37:55+01:00] INFO: Loading cookbooks [gitlab@0.0.1, package@0.1.0, logrotate@0.1.0, postgresql@0.1.0, redis@0.1.0, monitoring@0.1.0, registry@0.1.0, mattermost@0.1.0, consul@0.1.0, gitaly@0.1.0, praefect@0.1.0, gitlab-kas@0.1.0, gitlab-pages@0.1.0, letsencrypt@0.1.0, nginx@0.1.0, runit@5.1.7, acme@4.1.5, crond@0.1.0]
[2022-12-20T15:37:58+01:00] WARN: Selected systemd because /run/systemd/system/ exists
[2022-12-20T15:38:00+01:00] WARN: only_if block for templatesymlink[Create a gitlab_pages_secret and create a symlink to Rails root] returned a string, did you mean to run a command?
[2022-12-20T15:38:00+01:00] WARN: only_if block for templatesymlink[Create a gitlab_kas_secret and create a symlink to Rails root] returned a string, did you mean to run a command?
[2022-12-20T15:38:29+01:00] INFO: Running queued delayed notifications before re-raising exception
[2022-12-20T15:38:29+01:00] ERROR: Running exception handlers
[2022-12-20T15:38:29+01:00] ERROR: Exception handlers complete
[2022-12-20T15:38:29+01:00] FATAL: Stacktrace dumped to /opt/gitlab/embedded/cookbooks/cache/cinc-stacktrace.out
[2022-12-20T15:38:29+01:00] FATAL: ---------------------------------------------------------------------------------------
[2022-12-20T15:38:29+01:00] FATAL: PLEASE PROVIDE THE CONTENTS OF THE stacktrace.out FILE (above) IF YOU FILE A BUG REPORT
[2022-12-20T15:38:29+01:00] FATAL: ---------------------------------------------------------------------------------------
[2022-12-20T15:38:29+01:00] FATAL: Mixlib::ShellOut::ShellCommandFailed: rails_migration[gitlab-rails] (gitlab::database_migrations line 51) had an error: Mixlib::ShellOut::ShellCommandFailed: bash_hide_env[migrate gitlab-rails database] (gitlab::database_migrations line 18) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received ‘1’
---- Begin output of “bash” ----
STDOUT: rake aborted!
StandardError: An error has occurred, all later migrations canceled:
PG::ObjectNotInPrerequisiteState: ERROR: sequence must have same owner as table it is linked to
/opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20220902204048_move_security_findings_table_to_gitlab_partitions_dynamic_schema.rb:80:in block in up' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:123:in run_block’
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:134:in block in run_block_with_lock_timeout' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:129:in run_block_with_lock_timeout’
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:97:in run' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/lock_retries_helpers.rb:52:in with_lock_retries’
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/v2.rb:117:in with_lock_retries' /opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20220902204048_move_security_findings_table_to_gitlab_partitions_dynamic_schema.rb:63:in up’
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb:33:in block in exec_migration' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/query_analyzer.rb:37:in within’
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb:30:in exec_migration' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/lock_retry_mixin.rb:36:in ddl_transaction’
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:115:in configure_database' /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:95:in block (3 levels) in <top (required)>’
/opt/gitlab/embedded/bin/bundle:23:in load' /opt/gitlab/embedded/bin/bundle:23:in ’
Caused by:
ActiveRecord::StatementInvalid: PG::ObjectNotInPrerequisiteState: ERROR: sequence must have same owner as table it is linked to
/opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20220902204048_move_security_findings_table_to_gitlab_partitions_dynamic_schema.rb:80:in block in up' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:123:in run_block’
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:134:in block in run_block_with_lock_timeout' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:129:in run_block_with_lock_timeout’
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:97:in run' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/lock_retries_helpers.rb:52:in with_lock_retries’
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/v2.rb:117:in with_lock_retries' /opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20220902204048_move_security_findings_table_to_gitlab_partitions_dynamic_schema.rb:63:in up’
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb:33:in block in exec_migration' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/query_analyzer.rb:37:in within’
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb:30:in exec_migration' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/lock_retry_mixin.rb:36:in ddl_transaction’
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:115:in configure_database' /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:95:in block (3 levels) in <top (required)>’
/opt/gitlab/embedded/bin/bundle:23:in load' /opt/gitlab/embedded/bin/bundle:23:in ’
Caused by:
PG::ObjectNotInPrerequisiteState: ERROR: sequence must have same owner as table it is linked to
/opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20220902204048_move_security_findings_table_to_gitlab_partitions_dynamic_schema.rb:80:in block in up' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:123:in run_block’
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:134:in block in run_block_with_lock_timeout' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:129:in run_block_with_lock_timeout’
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:97:in run' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/lock_retries_helpers.rb:52:in with_lock_retries’
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/v2.rb:117:in with_lock_retries' /opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20220902204048_move_security_findings_table_to_gitlab_partitions_dynamic_schema.rb:63:in up’
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb:33:in block in exec_migration' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/query_analyzer.rb:37:in within’
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb:30:in exec_migration' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/lock_retry_mixin.rb:36:in ddl_transaction’
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:115:in configure_database' /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:95:in block (3 levels) in <top (required)>’
/opt/gitlab/embedded/bin/bundle:23:in load' /opt/gitlab/embedded/bin/bundle:23:in ’
Tasks: TOP => db:migrate
(See full trace by running task with --trace)
main: == 20220902204048 MoveSecurityFindingsTableToGitlabPartitionsDynamicSchema: migrating
main: – transaction_open?()
main: → 0.0000s
main: – execute(“LOCK TABLE vulnerability_scanners, security_scans, security_findings IN ACCESS EXCLUSIVE MODE\n”)
main: → 0.0008s
main: – execute(“SELECT\n pg_get_constraintdef(pg_catalog.pg_constraint.oid)\nFROM\n pg_catalog.pg_constraint\nINNER JOIN pg_class ON pg_class.oid = pg_catalog.pg_constraint.conrelid\nWHERE\n conname = ‘check_partition_number’ AND\n pg_class.relname = ‘security_findings’\n”)
main: → 0.0085s
main: – execute(“ALTER TABLE security_findings RENAME TO security_findings_1;\n”)
main: → 0.0007s
main: – execute(“ALTER INDEX security_findings_pkey RENAME TO security_findings_1_pkey;\n”)
main: → 0.0007s
main: – execute(“CREATE TABLE security_findings (\n LIKE security_findings_1 INCLUDING ALL\n) PARTITION BY LIST (partition_number);\n”)
main: → 0.0054s
main: – execute(“ALTER SEQUENCE security_findings_id_seq OWNED BY public.security_findings.id;\n”)
STDERR:
---- End output of “bash” ----
Ran “bash” returned 1