Hi all i have a failed Background Migration from a while a go that i am trying to fix.
System information
System: Ubuntu 20.04
Proxy: no
Current User: git
Using RVM: no
Ruby Version: 3.1.4p223
Gem Version: 3.4.22
Bundler Version:2.4.22
Rake Version: 13.0.6
Redis Version: 7.0.14
Sidekiq Version:6.5.12
Go Version: unknown
GitLab information
Version: 16.7.3-ee
Revision: 256ff8e8dae
Directory: /opt/gitlab/embedded/service/gitlab-rails
DB Adapter: PostgreSQL
DB Version: 13.12
URL: https://myddns:8050
HTTP Clone URL: https://myddns:8050/some-group/some-project.git
SSH Clone URL: git@myddns:some-group/some-project.git
Elasticsearch: no
Geo: no
Using LDAP: no
Using Omniauth: yes
Omniauth Providers:
GitLab Shell
Version: 14.32.0
Repository storages:
- default: unix:/var/opt/gitlab/gitaly/gitaly.socket
GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell
Gitaly
- default Address: unix:/var/opt/gitlab/gitaly/gitaly.socket
- default Version: 16.7.3
- default Git Version: 2.42.0
output
If i try and do
gitlab-rake gitlab:background_migrations:finalize[BackfillInternalOnNotes,notes,id,'[]']
I get
rake aborted!
ActiveRecord::StatementInvalid: PG::CheckViolation: ERROR: no partition of relation "batched_background_migration_job_transition_logs" found for row
DETAIL: Partition key of the failing row contains (created_at) = (2024-01-25 00:34:50.123161+00).
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `public_send'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `block in write_using_load_balancer'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:141:in `block in read_write'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:228:in `retry_with_backoff'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:130:in `read_write'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:126:in `write_using_load_balancer'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:61:in `block (2 levels) in <class:ConnectionProxy>'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `public_send'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `block in write_using_load_balancer'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:141:in `block in read_write'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:228:in `retry_with_backoff'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:130:in `read_write'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:126:in `write_using_load_balancer'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:78:in `transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `public_send'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `block in write_using_load_balancer'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:141:in `block in read_write'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:228:in `retry_with_backoff'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:130:in `read_write'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:126:in `write_using_load_balancer'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:78:in `transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database.rb:359:in `block in transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database.rb:358:in `transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/background_migration/batched_job.rb:84:in `block (2 levels) in <class:BatchedJob>'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `public_send'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `block in write_using_load_balancer'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:141:in `block in read_write'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:228:in `retry_with_backoff'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:130:in `read_write'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:126:in `write_using_load_balancer'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:78:in `transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database.rb:359:in `block in transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database.rb:358:in `transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/background_migration/batched_migration_wrapper.rb:34:in `rescue in perform'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/background_migration/batched_migration_wrapper.rb:28:in `perform'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/background_migration/batched_migration_runner.rb:30:in `run_migration_job'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/background_migration/batched_migration_runner.rb:140:in `run_migration_while'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/background_migration/batched_migration_runner.rb:80:in `finalize'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/background_migration/batched_migration_runner.rb:10:in `finalize'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/background_migrations.rake:72:in `finalize_migration'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/background_migrations.rake:18:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:25:in `load'
/opt/gitlab/embedded/bin/bundle:25:in `<main>'
What can I do to fix this?
I am still stuck on this issue, what else can i try?
If i run
sudo cat /opt/gitlab/embedded/cookbooks/cache/cinc-stacktrace.out
I get the following
Generated at 2024-02-22 12:58:16 +1100
Mixlib::ShellOut::ShellCommandFailed: rails_migration[gitlab-rails] (gitlab::database_migrations line 51) had an error: Mixlib::ShellOut::ShellCommandFailed: bash_hide_env[migrate gitlab-rails database] (gitlab::database_migrations line 20) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
---- Begin output of "bash" ----
STDOUT: rake aborted!
StandardError: An error has occurred, all later migrations canceled:
PG::UndefinedTable: ERROR: relation "gitlab_partitions_static.issue_search_data_00_search_vector_idx" does not exist
/opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20240105120320_disable_fastupdate_on_issue_search_data.rb:13:in `block (2 levels) in up'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:123:in `run_block'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:134:in `block in run_block_with_lock_timeout'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:129:in `run_block_with_lock_timeout'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:97:in `run'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/lock_retries_helpers.rb:52:in `with_lock_retries'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/v2.rb:99:in `with_lock_retries'
/opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20240105120320_disable_fastupdate_on_issue_search_data.rb:12:in `block in up'
/opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20240105120320_disable_fastupdate_on_issue_search_data.rb:34:in `block in each_index_name'
/opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20240105120320_disable_fastupdate_on_issue_search_data.rb:33:in `times'
/opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20240105120320_disable_fastupdate_on_issue_search_data.rb:33:in `each_index_name'
/opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20240105120320_disable_fastupdate_on_issue_search_data.rb:11:in `up'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb:33:in `block in exec_migration'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/query_analyzer.rb:40:in `within'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb:30:in `exec_migration'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/automatic_lock_writes_on_tables.rb:21:in `exec_migration'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/lock_retry_mixin.rb:36:in `ddl_transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/runner_backoff/active_record_mixin.rb:21:in `execute_migration_in_transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/pg_backend_pid.rb:15:in `block in with_advisory_lock_connection'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/pg_backend_pid.rb:12:in `with_advisory_lock_connection'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:138:in `configure_database'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:107:in `configure_pg_databases'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:94:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:25:in `load'
/opt/gitlab/embedded/bin/bundle:25:in `<main>'
Caused by:
ActiveRecord::StatementInvalid: PG::UndefinedTable: ERROR: relation "gitlab_partitions_static.issue_search_data_00_search_vector_idx" does not exist
/opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20240105120320_disable_fastupdate_on_issue_search_data.rb:13:in `block (2 levels) in up'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:123:in `run_block'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:134:in `block in run_block_with_lock_timeout'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:129:in `run_block_with_lock_timeout'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:97:in `run'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/lock_retries_helpers.rb:52:in `with_lock_retries'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/v2.rb:99:in `with_lock_retries'
/opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20240105120320_disable_fastupdate_on_issue_search_data.rb:12:in `block in up'
/opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20240105120320_disable_fastupdate_on_issue_search_data.rb:34:in `block in each_index_name'
/opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20240105120320_disable_fastupdate_on_issue_search_data.rb:33:in `times'
/opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20240105120320_disable_fastupdate_on_issue_search_data.rb:33:in `each_index_name'
/opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20240105120320_disable_fastupdate_on_issue_search_data.rb:11:in `up'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb:33:in `block in exec_migration'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/query_analyzer.rb:40:in `within'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb:30:in `exec_migration'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/automatic_lock_writes_on_tables.rb:21:in `exec_migration'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/lock_retry_mixin.rb:36:in `ddl_transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/runner_backoff/active_record_mixin.rb:21:in `execute_migration_in_transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/pg_backend_pid.rb:15:in `block in with_advisory_lock_connection'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/pg_backend_pid.rb:12:in `with_advisory_lock_connection'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:138:in `configure_database'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:107:in `configure_pg_databases'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:94:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:25:in `load'
/opt/gitlab/embedded/bin/bundle:25:in `<main>'
Caused by:
PG::UndefinedTable: ERROR: relation "gitlab_partitions_static.issue_search_data_00_search_vector_idx" does not exist
/opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20240105120320_disable_fastupdate_on_issue_search_data.rb:13:in `block (2 levels) in up'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:123:in `run_block'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:134:in `block in run_block_with_lock_timeout'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:129:in `run_block_with_lock_timeout'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:97:in `run'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/lock_retries_helpers.rb:52:in `with_lock_retries'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/v2.rb:99:in `with_lock_retries'
/opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20240105120320_disable_fastupdate_on_issue_search_data.rb:12:in `block in up'
/opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20240105120320_disable_fastupdate_on_issue_search_data.rb:34:in `block in each_index_name'
/opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20240105120320_disable_fastupdate_on_issue_search_data.rb:33:in `times'
/opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20240105120320_disable_fastupdate_on_issue_search_data.rb:33:in `each_index_name'
/opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20240105120320_disable_fastupdate_on_issue_search_data.rb:11:in `up'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb:33:in `block in exec_migration'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/query_analyzer.rb:40:in `within'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb:30:in `exec_migration'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/automatic_lock_writes_on_tables.rb:21:in `exec_migration'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/lock_retry_mixin.rb:36:in `ddl_transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/runner_backoff/active_record_mixin.rb:21:in `execute_migration_in_transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/pg_backend_pid.rb:15:in `block in with_advisory_lock_connection'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/pg_backend_pid.rb:12:in `with_advisory_lock_connection'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:138:in `configure_database'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:107:in `configure_pg_databases'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:94:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:25:in `load'
/opt/gitlab/embedded/bin/bundle:25:in `<main>'
Tasks: TOP => db:migrate
(See full trace by running task with --trace)
Running db:migrate rake task
main: == [advisory_lock_connection] object_id: 52780, pg_backend_pid: 1720139
main: == 20240105120320 DisableFastupdateOnIssueSearchData: migrating ===============
main: -- transaction_open?(nil)
main: -> 0.0000s
main: -- execute("ALTER INDEX gitlab_partitions_static.issue_search_data_00_search_vector_idx SET ( fastupdate = false ) ;\n")
main: == [advisory_lock_connection] object_id: 52780, pg_backend_pid: 1720139
STDERR:
---- End output of "bash" ----
Ran "bash" returned 1
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/mixlib-shellout-3.2.7/lib/mixlib/shellout.rb:300:in `invalid!'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/mixlib-shellout-3.2.7/lib/mixlib/shellout.rb:287:in `error!'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/mixlib-shellout-3.2.7/lib/mixlib/shellout/helper.rb:130:in `shell_out_compacted!'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/mixlib-shellout-3.2.7/lib/mixlib/shellout/helper.rb:54:in `shell_out!'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/provider/execute.rb:52:in `block (2 levels) in <class:Execute>'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/mixin/why_run.rb:51:in `add_action'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/provider.rb:293:in `converge_by'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/provider/execute.rb:50:in `block in <class:Execute>'
(eval):2:in `block in action_run'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/provider.rb:304:in `instance_eval'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/provider.rb:304:in `compile_and_converge_action'
(eval):2:in `action_run'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/provider.rb:245:in `run_action'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/resource.rb:601:in `block in run_action'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/resource.rb:628:in `with_umask'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/resource.rb:600:in `run_action'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/runner.rb:74:in `run_action'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/runner.rb:108:in `block in run_all_actions'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/runner.rb:108:in `each'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/runner.rb:108:in `run_all_actions'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/resource_collection.rb:64:in `insert'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/dsl/declare_resource.rb:267:in `declare_resource'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/dsl/resources.rb:36:in `bash_hide_env'
/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/resources/rails_migration.rb:20:in `block in class_from_file'
(eval):2:in `block in action_run'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/provider.rb:304:in `instance_eval'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/provider.rb:304:in `compile_and_converge_action'
(eval):2:in `action_run'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/provider.rb:245:in `run_action'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/resource.rb:601:in `block in run_action'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/resource.rb:628:in `with_umask'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/resource.rb:600:in `run_action'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/runner.rb:74:in `run_action'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/runner.rb:108:in `block in run_all_actions'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/runner.rb:108:in `each'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/runner.rb:108:in `run_all_actions'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/runner.rb:132:in `block in converge'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/resource_collection/resource_list.rb:96:in `block in execute_each_resource'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/resource_collection/stepable_iterator.rb:114:in `call_iterator_block'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/resource_collection/stepable_iterator.rb:85:in `step'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/resource_collection/stepable_iterator.rb:103:in `iterate'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/resource_collection/stepable_iterator.rb:54:in `each_with_index'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/resource_collection/resource_list.rb:94:in `execute_each_resource'
/opt/gitlab/embedded/lib/ruby/3.1.0/forwardable.rb:238:in `execute_each_resource'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/runner.rb:130:in `converge'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/client.rb:875:in `block in converge'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/client.rb:870:in `catch'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/client.rb:870:in `converge'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/client.rb:894:in `converge_and_save'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/client.rb:298:in `run'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/application.rb:305:in `run_with_graceful_exit_option'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/application.rb:281:in `block in run_chef_client'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/local_mode.rb:42:in `with_server_connectivity'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/application.rb:264:in `run_chef_client'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/application/base.rb:354:in `run_application'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/application.rb:67:in `run'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-bin-18.3.0/bin/cinc-client:25:in `<top (required)>'
/opt/gitlab/embedded/bin/cinc-client:25:in `load'
/opt/gitlab/embedded/bin/cinc-client:25:in `<main>'
>>>> Caused by Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
---- Begin output of "bash" ----
STDOUT: rake aborted!
StandardError: An error has occurred, all later migrations canceled:
PG::UndefinedTable: ERROR: relation "gitlab_partitions_static.issue_search_data_00_search_vector_idx" does not exist
/opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20240105120320_disable_fastupdate_on_issue_search_data.rb:13:in `block (2 levels) in up'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:123:in `run_block'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:134:in `block in run_block_with_lock_timeout'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:129:in `run_block_with_lock_timeout'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:97:in `run'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/lock_retries_helpers.rb:52:in `with_lock_retries'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/v2.rb:99:in `with_lock_retries'
/opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20240105120320_disable_fastupdate_on_issue_search_data.rb:12:in `block in up'
/opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20240105120320_disable_fastupdate_on_issue_search_data.rb:34:in `block in each_index_name'
/opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20240105120320_disable_fastupdate_on_issue_search_data.rb:33:in `times'
/opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20240105120320_disable_fastupdate_on_issue_search_data.rb:33:in `each_index_name'
/opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20240105120320_disable_fastupdate_on_issue_search_data.rb:11:in `up'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb:33:in `block in exec_migration'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/query_analyzer.rb:40:in `within'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb:30:in `exec_migration'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/automatic_lock_writes_on_tables.rb:21:in `exec_migration'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/lock_retry_mixin.rb:36:in `ddl_transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/runner_backoff/active_record_mixin.rb:21:in `execute_migration_in_transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/pg_backend_pid.rb:15:in `block in with_advisory_lock_connection'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/pg_backend_pid.rb:12:in `with_advisory_lock_connection'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:138:in `configure_database'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:107:in `configure_pg_databases'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:94:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:25:in `load'
/opt/gitlab/embedded/bin/bundle:25:in `<main>'
