Hey all - just attempted to update gitlab-ce (16.5.0-ce.0) through apt, getting this error when it runs gitlabctl reconfigure:
---- Begin output of "bash" ----
STDOUT: Running db:migrate rake task
main: == [advisory_lock_connection] object_id: 118660, pg_backend_pid: 2313756
main: == 20230918084159 DropColumnsFromGeoNodeStatusTable: migrating ================
main: -- remove_columns(:geo_node_statuses, :wikis_checksum_failed_count, :wikis_checksum_mismatch_count, :wikis_checksummed_count, :wikis_failed_count, :wikis_retrying_verification_count, :wikis_synced_count, :wikis_verification_failed_count, :wikis_verified_count, :design_repositories_count, :design_repositories_synced_count, :design_repositories_failed_count, :design_repositories_registry_count)
rake aborted!
StandardError: An error has occurred, this and all later migrations canceled:
PG::UndefinedColumn: ERROR: column "wikis_retrying_verification_count" of relation "geo_node_statuses" does not exist
Seems like it just wants to delete that column, so I’m not sure why it fails when it doesn’t exist. Anyway gitlab-ce install seems to be non-working as of now. I’ve tried specifically stopping all with gitlab-ctl stop and then running “apt upgrade” again but the error message is the same.
Generated at 2023-10-28 11:19:40 -0400
Mixlib::ShellOut::ShellCommandFailed: rails_migration[gitlab-rails] (gitlab::database_migrations line 51) had an error: Mixlib::ShellOut::ShellCommandFailed: bash_hide_env[migrate gitlab-rails database] (gitlab::database_migrations line 20) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
---- Begin output of "bash" ----
STDOUT: Running db:migrate rake task
main: == [advisory_lock_connection] object_id: 118660, pg_backend_pid: 2313756
main: == 20230918084159 DropColumnsFromGeoNodeStatusTable: migrating ================
main: -- remove_columns(:geo_node_statuses, :wikis_checksum_failed_count, :wikis_checksum_mismatch_count, :wikis_checksummed_count, :wikis_failed_count, :wikis_retrying_verification_count, :wikis_synced_count, :wikis_verification_failed_count, :wikis_verified_count, :design_repositories_count, :design_repositories_synced_count, :design_repositories_failed_count, :design_repositories_registry_count)
rake aborted!
StandardError: An error has occurred, this and all later migrations canceled:
PG::UndefinedColumn: ERROR: column "wikis_retrying_verification_count" of relation "geo_node_statuses" does not exist
/opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20230918084159_drop_columns_from_geo_node_status_table.rb:7:in `up'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb:33:in `block in exec_migration'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/query_analyzer.rb:37:in `within'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb:30:in `exec_migration'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/automatic_lock_writes_on_tables.rb:21:in `exec_migration'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:123:in `run_block'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:134:in `block in run_block_with_lock_timeout'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `public_send'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `block in write_using_load_balancer'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:137:in `block in read_write'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:224:in `retry_with_backoff'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:126:in `read_write'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:126:in `write_using_load_balancer'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:78:in `transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:129:in `run_block_with_lock_timeout'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:97:in `run'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/lock_retry_mixin.rb:34:in `ddl_transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/runner_backoff/active_record_mixin.rb:21:in `execute_migration_in_transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/pg_backend_pid.rb:15:in `block in with_advisory_lock_connection'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/pg_backend_pid.rb:12:in `with_advisory_lock_connection'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:123:in `configure_database'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:101:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:25:in `load'
/opt/gitlab/embedded/bin/bundle:25:in `<main>'
Caused by:
ActiveRecord::StatementInvalid: PG::UndefinedColumn: ERROR: column "wikis_retrying_verification_count" of relation "geo_node_statuses" does not exist
/opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20230918084159_drop_columns_from_geo_node_status_table.rb:7:in `up'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb:33:in `block in exec_migration'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/query_analyzer.rb:37:in `within'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb:30:in `exec_migration'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/automatic_lock_writes_on_tables.rb:21:in `exec_migration'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:123:in `run_block'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:134:in `block in run_block_with_lock_timeout'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `public_send'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `block in write_using_load_balancer'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:137:in `block in read_write'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:224:in `retry_with_backoff'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:126:in `read_write'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:126:in `write_using_load_balancer'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:78:in `transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:129:in `run_block_with_lock_timeout'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:97:in `run'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/lock_retry_mixin.rb:34:in `ddl_transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/runner_backoff/active_record_mixin.rb:21:in `execute_migration_in_transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/pg_backend_pid.rb:15:in `block in with_advisory_lock_connection'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/pg_backend_pid.rb:12:in `with_advisory_lock_connection'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:123:in `configure_database'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:101:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:25:in `load'
/opt/gitlab/embedded/bin/bundle:25:in `<main>'
Caused by:
PG::UndefinedColumn: ERROR: column "wikis_retrying_verification_count" of relation "geo_node_statuses" does not exist
/opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20230918084159_drop_columns_from_geo_node_status_table.rb:7:in `up'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb:33:in `block in exec_migration'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/query_analyzer.rb:37:in `within'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb:30:in `exec_migration'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/automatic_lock_writes_on_tables.rb:21:in `exec_migration'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:123:in `run_block'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:134:in `block in run_block_with_lock_timeout'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `public_send'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `block in write_using_load_balancer'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:137:in `block in read_write'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:224:in `retry_with_backoff'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:126:in `read_write'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:126:in `write_using_load_balancer'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:78:in `transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:129:in `run_block_with_lock_timeout'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:97:in `run'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/lock_retry_mixin.rb:34:in `ddl_transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/runner_backoff/active_record_mixin.rb:21:in `execute_migration_in_transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/pg_backend_pid.rb:15:in `block in with_advisory_lock_connection'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/pg_backend_pid.rb:12:in `with_advisory_lock_connection'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:123:in `configure_database'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:101:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:25:in `load'
/opt/gitlab/embedded/bin/bundle:25:in `<main>'
Tasks: TOP => db:migrate
(See full trace by running task with --trace)
STDERR:
---- End output of "bash" ----
Ran "bash" returned 1
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/mixlib-shellout-3.2.7/lib/mixlib/shellout.rb:300:in `invalid!'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/mixlib-shellout-3.2.7/lib/mixlib/shellout.rb:287:in `error!'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/mixlib-shellout-3.2.7/lib/mixlib/shellout/helper.rb:130:in `shell_out_compacted!'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/mixlib-shellout-3.2.7/lib/mixlib/shellout/helper.rb:54:in `shell_out!'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/provider/execute.rb:52:in `block (2 levels) in <class:Execute>'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/mixin/why_run.rb:51:in `add_action'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/provider.rb:293:in `converge_by'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/provider/execute.rb:50:in `block in <class:Execute>'
(eval):2:in `block in action_run'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/provider.rb:304:in `instance_eval'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/provider.rb:304:in `compile_and_converge_action'
(eval):2:in `action_run'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/provider.rb:245:in `run_action'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/resource.rb:601:in `block in run_action'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/resource.rb:628:in `with_umask'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/resource.rb:600:in `run_action'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/runner.rb:74:in `run_action'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/runner.rb:108:in `block in run_all_actions'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/runner.rb:108:in `each'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/runner.rb:108:in `run_all_actions'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/resource_collection.rb:64:in `insert'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/dsl/declare_resource.rb:267:in `declare_resource'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/dsl/resources.rb:36:in `bash_hide_env'
/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/resources/rails_migration.rb:20:in `block in class_from_file'
(eval):2:in `block in action_run'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/provider.rb:304:in `instance_eval'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/provider.rb:304:in `compile_and_converge_action'
(eval):2:in `action_run'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/provider.rb:245:in `run_action'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/resource.rb:601:in `block in run_action'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/resource.rb:628:in `with_umask'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/resource.rb:600:in `run_action'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/runner.rb:74:in `run_action'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/runner.rb:108:in `block in run_all_actions'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/runner.rb:108:in `each'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/runner.rb:108:in `run_all_actions'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/runner.rb:132:in `block in converge'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/resource_collection/resource_list.rb:96:in `block in execute_each_resource'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/resource_collection/stepable_iterator.rb:114:in `call_iterator_block'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/resource_collection/stepable_iterator.rb:85:in `step'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/resource_collection/stepable_iterator.rb:103:in `iterate'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/resource_collection/stepable_iterator.rb:54:in `each_with_index'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/resource_collection/resource_list.rb:94:in `execute_each_resource'
/opt/gitlab/embedded/lib/ruby/3.0.0/forwardable.rb:238:in `execute_each_resource'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/runner.rb:130:in `converge'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/client.rb:692:in `block in converge'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/client.rb:687:in `catch'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/client.rb:687:in `converge'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/client.rb:711:in `converge_and_save'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/client.rb:285:in `run'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/application.rb:305:in `run_with_graceful_exit_option'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/application.rb:281:in `block in run_chef_client'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/local_mode.rb:42:in `with_server_connectivity'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/application.rb:264:in `run_chef_client'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/application/base.rb:352:in `run_application'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/application.rb:67:in `run'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-bin-17.10.0/bin/cinc-client:25:in `<top (required)>'
/opt/gitlab/embedded/bin/cinc-client:25:in `load'
/opt/gitlab/embedded/bin/cinc-client:25:in `<main>'
>>>> Caused by Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
---- Begin output of "bash" ----
STDOUT: Running db:migrate rake task
main: == [advisory_lock_connection] object_id: 118660, pg_backend_pid: 2313756
main: == 20230918084159 DropColumnsFromGeoNodeStatusTable: migrating ================
main: -- remove_columns(:geo_node_statuses, :wikis_checksum_failed_count, :wikis_checksum_mismatch_count, :wikis_checksummed_count, :wikis_failed_count, :wikis_retrying_verification_count, :wikis_synced_count, :wikis_verification_failed_count, :wikis_verified_count, :design_repositories_count, :design_repositories_synced_count, :design_repositories_failed_count, :design_repositories_registry_count)
rake aborted!
StandardError: An error has occurred, this and all later migrations canceled:
PG::UndefinedColumn: ERROR: column "wikis_retrying_verification_count" of relation "geo_node_statuses" does not exist
/opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20230918084159_drop_columns_from_geo_node_status_table.rb:7:in `up'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb:33:in `block in exec_migration'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/query_analyzer.rb:37:in `within'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb:30:in `exec_migration'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/automatic_lock_writes_on_tables.rb:21:in `exec_migration'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:123:in `run_block'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:134:in `block in run_block_with_lock_timeout'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `public_send'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `block in write_using_load_balancer'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:137:in `block in read_write'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:224:in `retry_with_backoff'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:126:in `read_write'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:126:in `write_using_load_balancer'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:78:in `transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:129:in `run_block_with_lock_timeout'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:97:in `run'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/lock_retry_mixin.rb:34:in `ddl_transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/runner_backoff/active_record_mixin.rb:21:in `execute_migration_in_transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/pg_backend_pid.rb:15:in `block in with_advisory_lock_connection'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/pg_backend_pid.rb:12:in `with_advisory_lock_connection'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:123:in `configure_database'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:101:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:25:in `load'
/opt/gitlab/embedded/bin/bundle:25:in `<main>'
Caused by:
ActiveRecord::StatementInvalid: PG::UndefinedColumn: ERROR: column "wikis_retrying_verification_count" of relation "geo_node_statuses" does not exist
/opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20230918084159_drop_columns_from_geo_node_status_table.rb:7:in `up'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb:33:in `block in exec_migration'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/query_analyzer.rb:37:in `within'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb:30:in `exec_migration'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/automatic_lock_writes_on_tables.rb:21:in `exec_migration'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:123:in `run_block'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:134:in `block in run_block_with_lock_timeout'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `public_send'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `block in write_using_load_balancer'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:137:in `block in read_write'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:224:in `retry_with_backoff'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:126:in `read_write'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:126:in `write_using_load_balancer'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:78:in `transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:129:in `run_block_with_lock_timeout'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:97:in `run'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/lock_retry_mixin.rb:34:in `ddl_transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/runner_backoff/active_record_mixin.rb:21:in `execute_migration_in_transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/pg_backend_pid.rb:15:in `block in with_advisory_lock_connection'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/pg_backend_pid.rb:12:in `with_advisory_lock_connection'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:123:in `configure_database'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:101:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:25:in `load'
/opt/gitlab/embedded/bin/bundle:25:in `<main>'
Caused by:
PG::UndefinedColumn: ERROR: column "wikis_retrying_verification_count" of relation "geo_node_statuses" does not exist
/opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20230918084159_drop_columns_from_geo_node_status_table.rb:7:in `up'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb:33:in `block in exec_migration'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/query_analyzer.rb:37:in `within'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb:30:in `exec_migration'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/automatic_lock_writes_on_tables.rb:21:in `exec_migration'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:123:in `run_block'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:134:in `block in run_block_with_lock_timeout'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `public_send'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `block in write_using_load_balancer'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:137:in `block in read_write'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:224:in `retry_with_backoff'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:126:in `read_write'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:126:in `write_using_load_balancer'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:78:in `transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:129:in `run_block_with_lock_timeout'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:97:in `run'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/lock_retry_mixin.rb:34:in `ddl_transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/runner_backoff/active_record_mixin.rb:21:in `execute_migration_in_transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/pg_backend_pid.rb:15:in `block in with_advisory_lock_connection'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/pg_backend_pid.rb:12:in `with_advisory_lock_connection'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:123:in `configure_database'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:101:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:25:in `load'
/opt/gitlab/embedded/bin/bundle:25:in `<main>'
Tasks: TOP => db:migrate
(See full trace by running task with --trace)
STDERR:
---- End output of "bash" ----
Ran "bash" returned 1
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/mixlib-shellout-3.2.7/lib/mixlib/shellout.rb:300:in `invalid!'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/mixlib-shellout-3.2.7/lib/mixlib/shellout.rb:287:in `error!'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/mixlib-shellout-3.2.7/lib/mixlib/shellout/helper.rb:130:in `shell_out_compacted!'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/mixlib-shellout-3.2.7/lib/mixlib/shellout/helper.rb:54:in `shell_out!'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/provider/execute.rb:52:in `block (2 levels) in <class:Execute>'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/mixin/why_run.rb:51:in `add_action'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/provider.rb:293:in `converge_by'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/provider/execute.rb:50:in `block in <class:Execute>'
(eval):2:in `block in action_run'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/provider.rb:304:in `instance_eval'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/provider.rb:304:in `compile_and_converge_action'
(eval):2:in `action_run'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/provider.rb:245:in `run_action'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/resource.rb:601:in `block in run_action'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/resource.rb:628:in `with_umask'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/resource.rb:600:in `run_action'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/runner.rb:74:in `run_action'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/runner.rb:108:in `block in run_all_actions'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/runner.rb:108:in `each'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/runner.rb:108:in `run_all_actions'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/resource_collection.rb:64:in `insert'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/dsl/declare_resource.rb:267:in `declare_resource'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/dsl/resources.rb:36:in `bash_hide_env'
/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/resources/rails_migration.rb:20:in `block in class_from_file'
(eval):2:in `block in action_run'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/provider.rb:304:in `instance_eval'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/provider.rb:304:in `compile_and_converge_action'
(eval):2:in `action_run'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/provider.rb:245:in `run_action'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/resource.rb:601:in `block in run_action'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/resource.rb:628:in `with_umask'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/resource.rb:600:in `run_action'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/runner.rb:74:in `run_action'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/runner.rb:108:in `block in run_all_actions'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/runner.rb:108:in `each'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/runner.rb:108:in `run_all_actions'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/runner.rb:132:in `block in converge'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/resource_collection/resource_list.rb:96:in `block in execute_each_resource'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/resource_collection/stepable_iterator.rb:114:in `call_iterator_block'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/resource_collection/stepable_iterator.rb:85:in `step'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/resource_collection/stepable_iterator.rb:103:in `iterate'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/resource_collection/stepable_iterator.rb:54:in `each_with_index'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/resource_collection/resource_list.rb:94:in `execute_each_resource'
/opt/gitlab/embedded/lib/ruby/3.0.0/forwardable.rb:238:in `execute_each_resource'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/runner.rb:130:in `converge'