Hey all, I have gitlab ee installed via helm chart version 6.6.0 using the built in Postgressql subchart version 13.6.0. When i try and run a helm upgrade to Helm chart version 6.7.0 I start to see pods failing to deploy.
The first thing I check here is the gitlab-migratons pod for errors and see the following error.
Begin parsing .erb templates from /var/opt/gitlab/templates
Writing /srv/gitlab/config/cable.yml
Writing /srv/gitlab/config/database.yml
Writing /srv/gitlab/config/gitlab.yml
Writing /srv/gitlab/config/resque.yml
Begin parsing .tpl templates from /var/opt/gitlab/templates
Copying other config files found in /var/opt/gitlab/templates to /srv/gitlab/config
Attempting to run '/scripts/wait-for-deps /scripts/db-migrate' as a main process
Checking database migrations are up-to-date
Performing migrations (this will initialized if needed)
rake aborted!
StandardError: An error has occurred, this and all later migrations canceled:
PG::IndexCorrupted: ERROR: index "pg_depend_depender_index" contains unexpected zero page at block 43
HINT: Please REINDEX it.
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:49:in `exec'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:49:in `block (2 levels) in execute'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activesupport-6.1.7.2/lib/active_support/dependencies/interlock.rb:48:in `block in permit_concurrent_loads'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activesupport-6.1.7.2/lib/active_support/concurrency/share_lock.rb:187:in `yield_shares'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activesupport-6.1.7.2/lib/active_support/dependencies/interlock.rb:47:in `permit_concurrent_loads'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:48:in `block in execute'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:696:in `block (2 levels) in log'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activesupport-6.1.7.2/lib/active_support/concurrency/load_interlock_aware_monitor.rb:26:in `block (2 levels) in synchronize'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activesupport-6.1.7.2/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `handle_interrupt'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activesupport-6.1.7.2/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `block in synchronize'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activesupport-6.1.7.2/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `handle_interrupt'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activesupport-6.1.7.2/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `synchronize'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:695:in `block in log'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activesupport-6.1.7.2/lib/active_support/notifications/instrumenter.rb:24:in `instrument'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:687:in `log'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:47:in `execute'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/marginalia-1.11.1/lib/marginalia.rb:71:in `execute_with_marginalia'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.7.2/lib/active_record/connection_adapters/abstract/schema_statements.rb:322:in `create_table'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.7.2/lib/active_record/migration.rb:929:in `block in method_missing'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.7.2/lib/active_record/migration.rb:897:in `block in say_with_time'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.7.2/lib/active_record/migration.rb:897:in `say_with_time'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.7.2/lib/active_record/migration.rb:918:in `method_missing'
/srv/gitlab/lib/gitlab/database/migration_helpers/v2.rb:28:in `create_table'
/srv/gitlab/db/migrate/20221116160204_create_ml_experiment_metadata_and_ml_candidate_metadata.rb:5:in `change'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.7.2/lib/active_record/migration.rb:867:in `exec_migration'
/srv/gitlab/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb:33:in `block in exec_migration'
/srv/gitlab/lib/gitlab/database/query_analyzer.rb:37:in `within'
/srv/gitlab/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb:30:in `exec_migration'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.7.2/lib/active_record/migration.rb:851:in `block (2 levels) in migrate'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.7.2/lib/active_record/migration.rb:850:in `block in migrate'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.7.2/lib/active_record/connection_adapters/abstract/connection_pool.rb:462:in `with_connection'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.7.2/lib/active_record/migration.rb:849:in `migrate'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.7.2/lib/active_record/migration.rb:1037:in `migrate'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.7.2/lib/active_record/migration.rb:1329:in `block in execute_migration_in_transaction'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.7.2/lib/active_record/migration.rb:1380:in `block in ddl_transaction'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:320:in `block in transaction'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.7.2/lib/active_record/connection_adapters/abstract/transaction.rb:319:in `block in within_new_transaction'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activesupport-6.1.7.2/lib/active_support/concurrency/load_interlock_aware_monitor.rb:26:in `block (2 levels) in synchronize'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activesupport-6.1.7.2/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `handle_interrupt'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activesupport-6.1.7.2/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `block in synchronize'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activesupport-6.1.7.2/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `handle_interrupt'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activesupport-6.1.7.2/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `synchronize'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.7.2/lib/active_record/connection_adapters/abstract/transaction.rb:317:in `within_new_transaction'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:320:in `transaction'
/srv/gitlab/lib/gitlab/database/load_balancing/connection_proxy.rb:121:in `public_send'
/srv/gitlab/lib/gitlab/database/load_balancing/connection_proxy.rb:121:in `block in write_using_load_balancer'
/srv/gitlab/lib/gitlab/database/load_balancing/load_balancer.rb:127:in `block in read_write'
/srv/gitlab/lib/gitlab/database/load_balancing/load_balancer.rb:198:in `retry_with_backoff'
/srv/gitlab/lib/gitlab/database/load_balancing/load_balancer.rb:116:in `read_write'
/srv/gitlab/lib/gitlab/database/load_balancing/connection_proxy.rb:120:in `write_using_load_balancer'
/srv/gitlab/lib/gitlab/database/load_balancing/connection_proxy.rb:72:in `transaction'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.7.2/lib/active_record/transactions.rb:209:in `transaction'
/srv/gitlab/lib/gitlab/database.rb:369:in `block in transaction'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activesupport-6.1.7.2/lib/active_support/notifications.rb:205:in `instrument'
/srv/gitlab/lib/gitlab/database.rb:368:in `transaction'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.7.2/lib/active_record/migration.rb:1380:in `ddl_transaction'
/srv/gitlab/lib/gitlab/database/migrations/lock_retry_mixin.rb:36:in `ddl_transaction'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.7.2/lib/active_record/migration.rb:1328:in `execute_migration_in_transaction'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.7.2/lib/active_record/migration.rb:1302:in `each'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.7.2/lib/active_record/migration.rb:1302:in `migrate_without_lock'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.7.2/lib/active_record/migration.rb:1251:in `block in migrate'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.7.2/lib/active_record/migration.rb:1401:in `block in with_advisory_lock'
/srv/gitlab/lib/gitlab/database/migrations/pg_backend_pid.rb:15:in `block in with_advisory_lock_connection'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.7.2/lib/active_record/migration.rb:1416:in `block in with_advisory_lock_connection'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.7.2/lib/active_record/connection_adapters/abstract/connection_pool.rb:462:in `with_connection'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.7.2/lib/active_record/migration.rb:1416:in `with_advisory_lock_connection'
/srv/gitlab/lib/gitlab/database/migrations/pg_backend_pid.rb:12:in `with_advisory_lock_connection'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.7.2/lib/active_record/migration.rb:1397:in `with_advisory_lock'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.7.2/lib/active_record/migration.rb:1251:in `migrate'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.7.2/lib/active_record/migration.rb:1086:in `up'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.7.2/lib/active_record/migration.rb:1061:in `migrate'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.7.2/lib/active_record/tasks/database_tasks.rb:237:in `migrate'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.7.2/lib/active_record/railties/databases.rake:92:in `block (3 levels) in <main>'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.7.2/lib/active_record/railties/databases.rake:90:in `each'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.7.2/lib/active_record/railties/databases.rake:90:in `block (2 levels) in <main>'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/sentry-ruby-5.8.0/lib/sentry/rake.rb:26:in `execute'
/srv/gitlab/lib/tasks/gitlab/db.rake:117:in `configure_database'
/srv/gitlab/lib/tasks/gitlab/db.rake:95:in `block (3 levels) in <main>'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/sentry-ruby-5.8.0/lib/sentry/rake.rb:26:in `execute'
Caused by:
ActiveRecord::StatementInvalid: PG::IndexCorrupted: ERROR: index "pg_depend_depender_index" contains unexpected zero page at block 43
HINT: Please REINDEX it.
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:49:in `exec'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:49:in `block (2 levels) in execute'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activesupport-6.1.7.2/lib/active_support/dependencies/interlock.rb:48:in `block in permit_concurrent_loads'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activesupport-6.1.7.2/lib/active_support/concurrency/share_lock.rb:187:in `yield_shares'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activesupport-6.1.7.2/lib/active_support/dependencies/interlock.rb:47:in `permit_concurrent_loads'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:48:in `block in execute'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:696:in `block (2 levels) in log'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activesupport-6.1.7.2/lib/active_support/concurrency/load_interlock_aware_monitor.rb:26:in `block (2 levels) in synchronize'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activesupport-6.1.7.2/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `handle_interrupt'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activesupport-6.1.7.2/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `block in synchronize'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activesupport-6.1.7.2/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `handle_interrupt'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activesupport-6.1.7.2/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `synchronize'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:695:in `block in log'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activesupport-6.1.7.2/lib/active_support/notifications/instrumenter.rb:24:in `instrument'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:687:in `log'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:47:in `execute'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/marginalia-1.11.1/lib/marginalia.rb:71:in `execute_with_marginalia'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.7.2/lib/active_record/connection_adapters/abstract/schema_statements.rb:322:in `create_table'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.7.2/lib/active_record/migration.rb:929:in `block in method_missing'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.7.2/lib/active_record/migration.rb:897:in `block in say_with_time'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.7.2/lib/active_record/migration.rb:897:in `say_with_time'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.7.2/lib/active_record/migration.rb:918:in `method_missing'
/srv/gitlab/lib/gitlab/database/migration_helpers/v2.rb:28:in `create_table'
/srv/gitlab/db/migrate/20221116160204_create_ml_experiment_metadata_and_ml_candidate_metadata.rb:5:in `change'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.7.2/lib/active_record/migration.rb:867:in `exec_migration'
/srv/gitlab/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb:33:in `block in exec_migration'
/srv/gitlab/lib/gitlab/database/query_analyzer.rb:37:in `within'
/srv/gitlab/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb:30:in `exec_migration'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.7.2/lib/active_record/migration.rb:851:in `block (2 levels) in migrate'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.7.2/lib/active_record/migration.rb:850:in `block in migrate'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.7.2/lib/active_record/connection_adapters/abstract/connection_pool.rb:462:in `with_connection'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.7.2/lib/active_record/migration.rb:849:in `migrate'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.7.2/lib/active_record/migration.rb:1037:in `migrate'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.7.2/lib/active_record/migration.rb:1329:in `block in execute_migration_in_transaction'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.7.2/lib/active_record/migration.rb:1380:in `block in ddl_transaction'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:320:in `block in transaction'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.7.2/lib/active_record/connection_adapters/abstract/transaction.rb:319:in `block in within_new_transaction'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activesupport-6.1.7.2/lib/active_support/concurrency/load_interlock_aware_monitor.rb:26:in `block (2 levels) in synchronize'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activesupport-6.1.7.2/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `handle_interrupt'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activesupport-6.1.7.2/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `block in synchronize'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activesupport-6.1.7.2/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `handle_interrupt'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activesupport-6.1.7.2/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `synchronize'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.7.2/lib/active_record/connection_adapters/abstract/transaction.rb:317:in `within_new_transaction'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:320:in `transaction'
/srv/gitlab/lib/gitlab/database/load_balancing/connection_proxy.rb:121:in `public_send'
/srv/gitlab/lib/gitlab/database/load_balancing/connection_proxy.rb:121:in `block in write_using_load_balancer'
/srv/gitlab/lib/gitlab/database/load_balancing/load_balancer.rb:127:in `block in read_write'
/srv/gitlab/lib/gitlab/database/load_balancing/load_balancer.rb:198:in `retry_with_backoff'
/srv/gitlab/lib/gitlab/database/load_balancing/load_balancer.rb:116:in `read_write'
/srv/gitlab/lib/gitlab/database/load_balancing/connection_proxy.rb:120:in `write_using_load_balancer'
/srv/gitlab/lib/gitlab/database/load_balancing/connection_proxy.rb:72:in `transaction'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.7.2/lib/active_record/transactions.rb:209:in `transaction'
/srv/gitlab/lib/gitlab/database.rb:369:in `block in transaction'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activesupport-6.1.7.2/lib/active_support/notifications.rb:205:in `instrument'
/srv/gitlab/lib/gitlab/database.rb:368:in `transaction'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.7.2/lib/active_record/migration.rb:1380:in `ddl_transaction'
/srv/gitlab/lib/gitlab/database/migrations/lock_retry_mixin.rb:36:in `ddl_transaction'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.7.2/lib/active_record/migration.rb:1328:in `execute_migration_in_transaction'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.7.2/lib/active_record/migration.rb:1302:in `each'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.7.2/lib/active_record/migration.rb:1302:in `migrate_without_lock'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.7.2/lib/active_record/migration.rb:1251:in `block in migrate'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.7.2/lib/active_record/migration.rb:1401:in `block in with_advisory_lock'
/srv/gitlab/lib/gitlab/database/migrations/pg_backend_pid.rb:15:in `block in with_advisory_lock_connection'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.7.2/lib/active_record/migration.rb:1416:in `block in with_advisory_lock_connection'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.7.2/lib/active_record/connection_adapters/abstract/connection_pool.rb:462:in `with_connection'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.7.2/lib/active_record/migration.rb:1416:in `with_advisory_lock_connection'
/srv/gitlab/lib/gitlab/database/migrations/pg_backend_pid.rb:12:in `with_advisory_lock_connection'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.7.2/lib/active_record/migration.rb:1397:in `with_advisory_lock'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.7.2/lib/active_record/migration.rb:1251:in `migrate'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.7.2/lib/active_record/migration.rb:1086:in `up'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.7.2/lib/active_record/migration.rb:1061:in `migrate'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.7.2/lib/active_record/tasks/database_tasks.rb:237:in `migrate'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.7.2/lib/active_record/railties/databases.rake:92:in `block (3 levels) in <main>'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.7.2/lib/active_record/railties/databases.rake:90:in `each'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.7.2/lib/active_record/railties/databases.rake:90:in `block (2 levels) in <main>'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/sentry-ruby-5.8.0/lib/sentry/rake.rb:26:in `execute'
/srv/gitlab/lib/tasks/gitlab/db.rake:117:in `configure_database'
/srv/gitlab/lib/tasks/gitlab/db.rake:95:in `block (3 levels) in <main>'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/sentry-ruby-5.8.0/lib/sentry/rake.rb:26:in `execute'
Caused by:
PG::IndexCorrupted: ERROR: index "pg_depend_depender_index" contains unexpected zero page at block 43
HINT: Please REINDEX it.
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:49:in `exec'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:49:in `block (2 levels) in execute'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activesupport-6.1.7.2/lib/active_support/dependencies/interlock.rb:48:in `block in permit_concurrent_loads'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activesupport-6.1.7.2/lib/active_support/concurrency/share_lock.rb:187:in `yield_shares'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activesupport-6.1.7.2/lib/active_support/dependencies/interlock.rb:47:in `permit_concurrent_loads'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:48:in `block in execute'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:696:in `block (2 levels) in log'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activesupport-6.1.7.2/lib/active_support/concurrency/load_interlock_aware_monitor.rb:26:in `block (2 levels) in synchronize'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activesupport-6.1.7.2/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `handle_interrupt'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activesupport-6.1.7.2/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `block in synchronize'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activesupport-6.1.7.2/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `handle_interrupt'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activesupport-6.1.7.2/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `synchronize'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:695:in `block in log'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activesupport-6.1.7.2/lib/active_support/notifications/instrumenter.rb:24:in `instrument'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:687:in `log'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:47:in `execute'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/marginalia-1.11.1/lib/marginalia.rb:71:in `execute_with_marginalia'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.7.2/lib/active_record/connection_adapters/abstract/schema_statements.rb:322:in `create_table'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.7.2/lib/active_record/migration.rb:929:in `block in method_missing'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.7.2/lib/active_record/migration.rb:897:in `block in say_with_time'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.7.2/lib/active_record/migration.rb:897:in `say_with_time'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.7.2/lib/active_record/migration.rb:918:in `method_missing'
/srv/gitlab/lib/gitlab/database/migration_helpers/v2.rb:28:in `create_table'
/srv/gitlab/db/migrate/20221116160204_create_ml_experiment_metadata_and_ml_candidate_metadata.rb:5:in `change'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.7.2/lib/active_record/migration.rb:867:in `exec_migration'
/srv/gitlab/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb:33:in `block in exec_migration'
/srv/gitlab/lib/gitlab/database/query_analyzer.rb:37:in `within'
/srv/gitlab/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb:30:in `exec_migration'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.7.2/lib/active_record/migration.rb:851:in `block (2 levels) in migrate'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.7.2/lib/active_record/migration.rb:850:in `block in migrate'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.7.2/lib/active_record/connection_adapters/abstract/connection_pool.rb:462:in `with_connection'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.7.2/lib/active_record/migration.rb:849:in `migrate'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.7.2/lib/active_record/migration.rb:1037:in `migrate'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.7.2/lib/active_record/migration.rb:1329:in `block in execute_migration_in_transaction'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.7.2/lib/active_record/migration.rb:1380:in `block in ddl_transaction'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:320:in `block in transaction'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.7.2/lib/active_record/connection_adapters/abstract/transaction.rb:319:in `block in within_new_transaction'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activesupport-6.1.7.2/lib/active_support/concurrency/load_interlock_aware_monitor.rb:26:in `block (2 levels) in synchronize'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activesupport-6.1.7.2/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `handle_interrupt'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activesupport-6.1.7.2/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `block in synchronize'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activesupport-6.1.7.2/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `handle_interrupt'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activesupport-6.1.7.2/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `synchronize'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.7.2/lib/active_record/connection_adapters/abstract/transaction.rb:317:in `within_new_transaction'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:320:in `transaction'
/srv/gitlab/lib/gitlab/database/load_balancing/connection_proxy.rb:121:in `public_send'
/srv/gitlab/lib/gitlab/database/load_balancing/connection_proxy.rb:121:in `block in write_using_load_balancer'
/srv/gitlab/lib/gitlab/database/load_balancing/load_balancer.rb:127:in `block in read_write'
/srv/gitlab/lib/gitlab/database/load_balancing/load_balancer.rb:198:in `retry_with_backoff'
/srv/gitlab/lib/gitlab/database/load_balancing/load_balancer.rb:116:in `read_write'
/srv/gitlab/lib/gitlab/database/load_balancing/connection_proxy.rb:120:in `write_using_load_balancer'
/srv/gitlab/lib/gitlab/database/load_balancing/connection_proxy.rb:72:in `transaction'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.7.2/lib/active_record/transactions.rb:209:in `transaction'
/srv/gitlab/lib/gitlab/database.rb:369:in `block in transaction'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activesupport-6.1.7.2/lib/active_support/notifications.rb:205:in `instrument'
/srv/gitlab/lib/gitlab/database.rb:368:in `transaction'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.7.2/lib/active_record/migration.rb:1380:in `ddl_transaction'
/srv/gitlab/lib/gitlab/database/migrations/lock_retry_mixin.rb:36:in `ddl_transaction'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.7.2/lib/active_record/migration.rb:1328:in `execute_migration_in_transaction'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.7.2/lib/active_record/migration.rb:1302:in `each'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.7.2/lib/active_record/migration.rb:1302:in `migrate_without_lock'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.7.2/lib/active_record/migration.rb:1251:in `block in migrate'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.7.2/lib/active_record/migration.rb:1401:in `block in with_advisory_lock'
/srv/gitlab/lib/gitlab/database/migrations/pg_backend_pid.rb:15:in `block in with_advisory_lock_connection'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.7.2/lib/active_record/migration.rb:1416:in `block in with_advisory_lock_connection'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.7.2/lib/active_record/connection_adapters/abstract/connection_pool.rb:462:in `with_connection'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.7.2/lib/active_record/migration.rb:1416:in `with_advisory_lock_connection'
/srv/gitlab/lib/gitlab/database/migrations/pg_backend_pid.rb:12:in `with_advisory_lock_connection'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.7.2/lib/active_record/migration.rb:1397:in `with_advisory_lock'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.7.2/lib/active_record/migration.rb:1251:in `migrate'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.7.2/lib/active_record/migration.rb:1086:in `up'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.7.2/lib/active_record/migration.rb:1061:in `migrate'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.7.2/lib/active_record/tasks/database_tasks.rb:237:in `migrate'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.7.2/lib/active_record/railties/databases.rake:92:in `block (3 levels) in <main>'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.7.2/lib/active_record/railties/databases.rake:90:in `each'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.7.2/lib/active_record/railties/databases.rake:90:in `block (2 levels) in <main>'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/sentry-ruby-5.8.0/lib/sentry/rake.rb:26:in `execute'
/srv/gitlab/lib/tasks/gitlab/db.rake:117:in `configure_database'
/srv/gitlab/lib/tasks/gitlab/db.rake:95:in `block (3 levels) in <main>'
/srv/gitlab/vendor/bundle/ruby/3.0.0/gems/sentry-ruby-5.8.0/lib/sentry/rake.rb:26:in `execute'
Tasks: TOP => db:migrate
(See full trace by running task with --trace)
Running db:migrate rake task
main: == [advisory_lock_connection] object_id: 220640, pg_backend_pid: 1762257
main: == 20221116160204 CreateMlExperimentMetadataAndMlCandidateMetadata: migrating =
main: -- create_table(:ml_experiment_metadata)
main: -- quote_column_name(:name)
main: -> 0.0000s
main: -- quote_column_name(:value)
main: -> 0.0000s
Has anyone seem similar errors to this before or have any hints on where to begin troubleshooting this issue? Thanks.