Upgrade Failed 17.12 → 17.20 at database_migrations in Docker
- Container always restart at here : Recipe: gitlab::database_migrations
Recipe: gitlab::database_migrations
* ruby_block[check remote PG version] action nothing (skipped due to action :nothing)
* rails_migration[gitlab-rails] action run[2024-07-19T01:20:30+00:00] WARN: gitlab-rails does not have a log_group or default logdir mode defined. Setting to 0700.
- there are too many ERROR with PG like below
StandardError: An error has occurred, this and all later migrations canceled:
PG::DependentObjectsStillExist: ERROR: cannot drop desired object(s) because other objects depend on them
DETAIL: default value for column id of table vulnerability_feedback depends on sequence vulnerability_feedback_id_seq
HINT: Use DROP ... CASCADE to drop the dependent objects too.
Whole log file: to see here: Upgrade Failed 17.12 -> 17.20 at database_migrations in Docker (#473337) · Issues · GitLab.org / GitLab · GitLab
Configuration
*there was OK before by 17.12 image And No docker run command change between using 17.12 or 17.20 Image
*Now I rollback to 17.12. Only by change the image and 17.12 seems OK ,god bless 
Versions
Please select whether options apply, and add the version information.
- Docker: Standalone 23.0.1
- Docker image: gitlab/gitlab-ce:17.20-ce.0
some link
4 Likes
I have the same issue, but with a different object.
PG::DependentObjectsStillExist: ERROR: cannot drop desired object(s) because other objects depend on them
DETAIL: default value for column id of table merge_request_assignees depends on sequence merge_request_assignees_id_seq
HINT: Use DROP ... CASCADE to drop the dependent objects too.
The system is now bricked and I do not know how to roll back, and if it is safe.
I updated gitlab-runner and gitlab-ce (omnibus) it via aptitude from the Debian repo.
Rolling back is in general not a good idea because we need to update every software sooner or later.
How can this problem be solved?
EDIT: I created a ticket GitLab CE omnibus bricked after upgrade (PG::DependentObjectsStillExist) (#473406) · Issues · GitLab.org / GitLab · GitLab
1 Like
I see some issues about this a problem with latatest upgrade process.
GitLab v17.1.2-ee
GitLab Shell 14.36.0
GitLab Workhorse v17.1.2
GitLab API v4
GitLab KAS 17.1.2
Ruby 3.1.5p253
Rails 7.0.8.4
PostgreSQL (main) 14.11
PostgreSQL (ci) 14.11
Redis
---- End output of "bash" ----
Ran "bash" returned 1
Running handlers complete
[2024-07-22T08:21:42+00:00] ERROR: Exception handlers complete
Infra Phase failed. 0 resources updated in 39 seconds
[2024-07-22T08:21:42+00:00] FATAL: Stacktrace dumped to /opt/gitlab/embedded/cookbooks/cache/cinc-stacktrace.out
[2024-07-22T08:21:42+00:00] FATAL: ---------------------------------------------------------------------------------------
[2024-07-22T08:21:42+00:00] FATAL: PLEASE PROVIDE THE CONTENTS OF THE stacktrace.out FILE (above) IF YOU FILE A BUG REPORT
[2024-07-22T08:21:42+00:00] FATAL: ---------------------------------------------------------------------------------------
[2024-07-22T08:21:42+00:00] FATAL: Mixlib::ShellOut::ShellCommandFailed: rails_migration[gitlab-rails] (gitlab::database_migrations line 51) had an error: Mixlib::ShellOut::ShellCommandFailed: bash_hide_env[migrate gitlab-rails database] (gitlab::database_migrations line 20) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
---- Begin output of "bash" ----
STDOUT: rake aborted!
StandardError: An error has occurred, this and all later migrations canceled:
PG::DependentObjectsStillExist: ERROR: cannot drop desired object(s) because other objects depend on them
DETAIL: default value for column id of table protected_branches depends on sequence protected_branches_id_seq
HINT: Use DROP ... CASCADE to drop the dependent objects too.
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:776:in `block in revert_initialize_conversion_of_integer_to_bigint'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:776:in `each'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:776:in `revert_initialize_conversion_of_integer_to_bigint'
/opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20240709014310_cleanup_bigint_conversions_for_p_ci_builds_attempt2.rb:28:in `up'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb:33:in `block in exec_migration'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/query_analyzer.rb:40:in `within'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb:30:in `exec_migration'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/automatic_lock_writes_on_tables.rb:21:in `exec_migration'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:123:in `run_block'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:134:in `block in run_block_with_lock_timeout'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `public_send'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `block in write_using_load_balancer'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:141:in `block in read_write'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:228:in `retry_with_backoff'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:130:in `read_write'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:126:in `write_using_load_balancer'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:78:in `transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:129:in `run_block_with_lock_timeout'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:97:in `run'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/lock_retry_mixin.rb:52:in `ddl_transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/runner_backoff/active_record_mixin.rb:21:in `execute_migration_in_transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/pg_backend_pid.rb:15:in `block in with_advisory_lock_connection'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/pg_backend_pid.rb:12:in `with_advisory_lock_connection'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:136:in `configure_database'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:107:in `configure_pg_databases'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:94:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:25:in `load'
/opt/gitlab/embedded/bin/bundle:25:in `<main>'
Caused by:
ActiveRecord::StatementInvalid: PG::DependentObjectsStillExist: ERROR: cannot drop desired object(s) because other objects depend on them
DETAIL: default value for column id of table protected_branches depends on sequence protected_branches_id_seq
HINT: Use DROP ... CASCADE to drop the dependent objects too.
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:776:in `block in revert_initialize_conversion_of_integer_to_bigint'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:776:in `each'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:776:in `revert_initialize_conversion_of_integer_to_bigint'
/opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20240709014310_cleanup_bigint_conversions_for_p_ci_builds_attempt2.rb:28:in `up'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb:33:in `block in exec_migration'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/query_analyzer.rb:40:in `within'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb:30:in `exec_migration'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/automatic_lock_writes_on_tables.rb:21:in `exec_migration'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:123:in `run_block'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:134:in `block in run_block_with_lock_timeout'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `public_send'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `block in write_using_load_balancer'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:141:in `block in read_write'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:228:in `retry_with_backoff'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:130:in `read_write'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:126:in `write_using_load_balancer'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:78:in `transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:129:in `run_block_with_lock_timeout'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:97:in `run'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/lock_retry_mixin.rb:52:in `ddl_transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/runner_backoff/active_record_mixin.rb:21:in `execute_migration_in_transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/pg_backend_pid.rb:15:in `block in with_advisory_lock_connection'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/pg_backend_pid.rb:12:in `with_advisory_lock_connection'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:136:in `configure_database'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:107:in `configure_pg_databases'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:94:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:25:in `load'
/opt/gitlab/embedded/bin/bundle:25:in `<main>'
Caused by:
PG::DependentObjectsStillExist: ERROR: cannot drop desired object(s) because other objects depend on them
DETAIL: default value for column id of table protected_branches depends on sequence protected_branches_id_seq
HINT: Use DROP ... CASCADE to drop the dependent objects too.
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:776:in `block in revert_initialize_conversion_of_integer_to_bigint'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:776:in `each'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:776:in `revert_initialize_conversion_of_integer_to_bigint'
/opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20240709014310_cleanup_bigint_conversions_for_p_ci_builds_attempt2.rb:28:in `up'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb:33:in `block in exec_migration'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/query_analyzer.rb:40:in `within'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb:30:in `exec_migration'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/automatic_lock_writes_on_tables.rb:21:in `exec_migration'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:123:in `run_block'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:134:in `block in run_block_with_lock_timeout'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `public_send'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `block in write_using_load_balancer'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:141:in `block in read_write'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:228:in `retry_with_backoff'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:130:in `read_write'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:126:in `write_using_load_balancer'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:78:in `transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:129:in `run_block_with_lock_timeout'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:97:in `run'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/lock_retry_mixin.rb:52:in `ddl_transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/runner_backoff/active_record_mixin.rb:21:in `execute_migration_in_transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/pg_backend_pid.rb:15:in `block in with_advisory_lock_connection'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/pg_backend_pid.rb:12:in `with_advisory_lock_connection'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:136:in `configure_database'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:107:in `configure_pg_databases'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:94:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:25:in `load'
/opt/gitlab/embedded/bin/bundle:25:in `<main>'
Tasks: TOP => db:migrate
(See full trace by running task with --trace)
Running db:migrate rake task
main: == [advisory_lock_connection] object_id: 57080, pg_backend_pid: 19686
main: == 20240709014310 CleanupBigintConversionsForPCiBuildsAttempt2: migrating =====
main: -- current_schema(nil)
main: -> 0.0011s
main: -- column_exists?(:p_ci_builds, "auto_canceled_by_id_convert_to_bigint")
main: -> 0.0057s
main: -- column_exists?(:p_ci_builds, "commit_id_convert_to_bigint")
main: -> 0.0038s
main: -- column_exists?(:p_ci_builds, "erased_by_id_convert_to_bigint")
main: -> 0.0039s
main: -- column_exists?(:p_ci_builds, "project_id_convert_to_bigint")
main: -> 0.0029s
main: -- column_exists?(:p_ci_builds, "runner_id_convert_to_bigint")
main: -> 0.0027s
main: -- column_exists?(:p_ci_builds, "trigger_request_id_convert_to_bigint")
main: -> 0.0030s
main: -- column_exists?(:p_ci_builds, "upstream_pipeline_id_convert_to_bigint")
main: -> 0.0028s
main: -- column_exists?(:p_ci_builds, "user_id_convert_to_bigint")
main: -> 0.0029s
main: -- quote_table_name(:p_ci_builds)
main: -> 0.0000s
main: -- execute("LOCK TABLE ONLY \"p_ci_builds\" IN ACCESS EXCLUSIVE MODE")
main: -> 0.0005s
main: -- quote_table_name(:p_ci_builds)
main: -> 0.0000s
main: -- execute("LOCK TABLE \"p_ci_builds\" IN ACCESS EXCLUSIVE MODE")
main: -> 0.0004s
main: -- remove_column(:p_ci_builds, "auto_canceled_by_id_convert_to_bigint", {:if_exists=>true})
main: -> 0.0046s
main: -- remove_column(:p_ci_builds, "commit_id_convert_to_bigint", {:if_exists=>true})
main: == [advisory_lock_connection] object_id: 57080, pg_backend_pid: 19686
STDERR:
---- End output of "bash" ----
Ran "bash" returned 1
I can not upoload /opt/gitlab/embedded/cookbooks/cache/cinc-stacktrace.outbecused are in production, and whe are in work times. It is de frist time that fails and upgrade.
That’s why I was so bold to upgrade first thing in the morning.
I HAVE also tried using gitlab::database_migrations and another one that I don’t remember now. At that moment I recovered the snapshot that I always take before updating.
As my environment is docker, I rollback through by change the image and recreate.