PostgreSQL down after upgrade from 13.10 to 13.12

The command I used to upgrade from 13.10 to 13.12

docker-compose pull
docker-compose up -d

Problem

PostgreSQL bootloop and log: database system was interrupted;

Because it’s a minor version update, I was have no backup :face_with_thermometer:

Solution I try

Fix the interrupted by follow command:

pg_resetwal /var/opt/gitlab/postgresql/data

Fallback to 13.10 still not working, Http request return 500 and the log loop these two messages:

2021-08-12T08:23:07.747Z: Cached record for ApplicationSetting couldn't be loaded, falling back to uncached record: PG::UndefinedTable: ERROR:  relation "application_settings" does not exist
LINE 1: ...on:sidekiq*/ SELECT "application_settings".* FROM "applicati...
                                                             ^

2021-08-12T08:23:14.053Z: Cached record for ApplicationSetting couldn't be loaded, falling back to uncached record: PG::UndefinedTable: ERROR:  relation "application_settings" does not exist
LINE 1: ...cation:web*/ SELECT "application_settings".* FROM "applicati...
                                                             
==> /var/log/gitlab/sidekiq/current <==
PG::UndefinedTable: ERROR:  relation "application_settings" does not exist
LINE 1: ...on:sidekiq*/ SELECT "application_settings".* FROM "applicati...
                                                             ^
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.0.3.4/lib/active_record/connection_adapters/postgresql_adapter.rb:675:in `exec_params'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.0.3.4/lib/active_record/connection_adapters/postgresql_adapter.rb:675:in `block (2 levels) in exec_no_cache'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activesupport-6.0.3.4/lib/active_support/dependencies/interlock.rb:48:in `block in permit_concurrent_loads'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activesupport-6.0.3.4/lib/active_support/concurrency/share_lock.rb:187:in `yield_shares'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activesupport-6.0.3.4/lib/active_support/dependencies/interlock.rb:47:in `permit_concurrent_loads'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.0.3.4/lib/active_record/connection_adapters/postgresql_adapter.rb:674:in `block in exec_no_cache'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.0.3.4/lib/active_record/connection_adapters/abstract_adapter.rb:722:in `block (2 levels) in log'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activesupport-6.0.3.4/lib/active_support/concurrency/load_interlock_aware_monitor.rb:26:in `block (2 levels) in synchronize'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activesupport-6.0.3.4/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `handle_interrupt'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activesupport-6.0.3.4/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `block in synchronize'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activesupport-6.0.3.4/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `handle_interrupt'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activesupport-6.0.3.4/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `synchronize'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.0.3.4/lib/active_record/connection_adapters/abstract_adapter.rb:721:in `block in log'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activesupport-6.0.3.4/lib/active_support/notifications/instrumenter.rb:24:in `instrument'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.0.3.4/lib/active_record/connection_adapters/abstract_adapter.rb:712:in `log'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.0.3.4/lib/active_record/connection_adapters/postgresql_adapter.rb:673:in `exec_no_cache'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.0.3.4/lib/active_record/connection_adapters/postgresql_adapter.rb:654:in `execute_and_clear'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/marginalia-1.10.0/lib/marginalia.rb:95:in `execute_and_clear_with_marginalia'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.0.3.4/lib/active_record/connection_adapters/postgresql/database_statements.rb:98:in `exec_query'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.0.3.4/lib/active_record/connection_adapters/abstract/database_statements.rb:489:in `select'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.0.3.4/lib/active_record/connection_adapters/abstract/database_statements.rb:70:in `select_all'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.0.3.4/lib/active_record/connection_adapters/abstract/query_cache.rb:107:in `select_all'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.0.3.4/lib/active_record/querying.rb:46:in `find_by_sql'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.0.3.4/lib/active_record/relation.rb:821:in `block in exec_queries'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.0.3.4/lib/active_record/relation.rb:839:in `skip_query_cache_if_necessary'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.0.3.4/lib/active_record/relation.rb:808:in `exec_queries'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.0.3.4/lib/active_record/relation.rb:626:in `load'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.0.3.4/lib/active_record/relation.rb:250:in `records'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.0.3.4/lib/active_record/relation.rb:245:in `to_ary'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.0.3.4/lib/active_record/relation/finder_methods.rb:527:in `find_nth_with_limit'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.0.3.4/lib/active_record/relation/finder_methods.rb:512:in `find_nth'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.0.3.4/lib/active_record/relation/finder_methods.rb:120:in `first'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.0.3.4/lib/active_record/relation/finder_methods.rb:151:in `last'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.0.3.4/lib/active_record/querying.rb:21:in `last'
/opt/gitlab/embedded/service/gitlab-rails/app/models/concerns/cacheable_attributes.rb:19:in `current_without_cache'
/opt/gitlab/embedded/service/gitlab-rails/app/models/concerns/cacheable_attributes.rb:63:in `rescue in current'
/opt/gitlab/embedded/service/gitlab-rails/app/models/concerns/cacheable_attributes.rb:51:in `current'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/current_settings.rb:56:in `uncached_application_settings'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/current_settings.rb:38:in `ensure_application_settings!'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/current_settings.rb:11:in `block in current_application_settings'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/null_request_store.rb:34:in `fetch'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/safe_request_store.rb:12:in `fetch'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/current_settings.rb:11:in `current_application_settings'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/current_settings.rb:28:in `method_missing'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/prometheus.rb:100:in `prometheus_metrics_enabled_unmemoized'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/prometheus.rb:26:in `block in prometheus_metrics_enabled?'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/utils/strong_memoize.rb:30:in `strong_memoize'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/prometheus.rb:25:in `prometheus_metrics_enabled?'
/opt/gitlab/embedded/service/gitlab-rails/config/initializers/7_prometheus_metrics.rb:43:in `<top (required)>'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activesupport-6.0.3.4/lib/active_support/dependencies.rb:318:in `load'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activesupport-6.0.3.4/lib/active_support/dependencies.rb:318:in `block in load'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activesupport-6.0.3.4/lib/active_support/dependencies.rb:291:in `load_dependency'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activesupport-6.0.3.4/lib/active_support/dependencies.rb:318:in `load'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/railties-6.0.3.4/lib/rails/engine.rb:666:in `block in load_config_initializer'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activesupport-6.0.3.4/lib/active_support/notifications.rb:182:in `instrument'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/railties-6.0.3.4/lib/rails/engine.rb:665:in `load_config_initializer'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/railties-6.0.3.4/lib/rails/engine.rb:625:in `block (2 levels) in <class:Engine>'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/railties-6.0.3.4/lib/rails/engine.rb:624:in `each'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/railties-6.0.3.4/lib/rails/engine.rb:624:in `block in <class:Engine>'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/railties-6.0.3.4/lib/rails/initializable.rb:32:in `instance_exec'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/railties-6.0.3.4/lib/rails/initializable.rb:32:in `run'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/railties-6.0.3.4/lib/rails/initializable.rb:61:in `block in run_initializers'
/opt/gitlab/embedded/lib/ruby/2.7.0/tsort.rb:228:in `block in tsort_each'
/opt/gitlab/embedded/lib/ruby/2.7.0/tsort.rb:350:in `block (2 levels) in each_strongly_connected_component'
/opt/gitlab/embedded/lib/ruby/2.7.0/tsort.rb:422:in `block (2 levels) in each_strongly_connected_component_from'
/opt/gitlab/embedded/lib/ruby/2.7.0/tsort.rb:431:in `each_strongly_connected_component_from'
/opt/gitlab/embedded/lib/ruby/2.7.0/tsort.rb:421:in `block in each_strongly_connected_component_from'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/railties-6.0.3.4/lib/rails/initializable.rb:50:in `each'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/railties-6.0.3.4/lib/rails/initializable.rb:50:in `tsort_each_child'
/opt/gitlab/embedded/lib/ruby/2.7.0/tsort.rb:415:in `call'
/opt/gitlab/embedded/lib/ruby/2.7.0/tsort.rb:415:in `each_strongly_connected_component_from'
/opt/gitlab/embedded/lib/ruby/2.7.0/tsort.rb:349:in `block in each_strongly_connected_component'
/opt/gitlab/embedded/lib/ruby/2.7.0/tsort.rb:347:in `each'
/opt/gitlab/embedded/lib/ruby/2.7.0/tsort.rb:347:in `call'
/opt/gitlab/embedded/lib/ruby/2.7.0/tsort.rb:347:in `each_strongly_connected_component'
/opt/gitlab/embedded/lib/ruby/2.7.0/tsort.rb:226:in `tsort_each'
/opt/gitlab/embedded/lib/ruby/2.7.0/tsort.rb:205:in `tsort_each'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/railties-6.0.3.4/lib/rails/initializable.rb:60:in `run_initializers'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/railties-6.0.3.4/lib/rails/application.rb:363:in `initialize!'
/opt/gitlab/embedded/service/gitlab-rails/config/environment.rb:7:in `<top (required)>'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activesupport-6.0.3.4/lib/active_support/dependencies.rb:324:in `require'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activesupport-6.0.3.4/lib/active_support/dependencies.rb:324:in `block in require'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activesupport-6.0.3.4/lib/active_support/dependencies.rb:291:in `load_dependency'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activesupport-6.0.3.4/lib/active_support/dependencies.rb:324:in `require'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/sidekiq-5.2.9/lib/sidekiq/cli.rb:288:in `boot_system'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/sidekiq-5.2.9/lib/sidekiq/cli.rb:46:in `run'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/sidekiq-5.2.9/bin/sidekiq:12:in `<top (required)>'
/opt/gitlab/embedded/bin/sidekiq:23:in `load'
/opt/gitlab/embedded/bin/sidekiq:23:in `<top (required)>'
/opt/gitlab/embedded/lib/ruby/site_ruby/2.7.0/bundler/cli/exec.rb:63:in `load'
/opt/gitlab/embedded/lib/ruby/site_ruby/2.7.0/bundler/cli/exec.rb:63:in `kernel_load'
/opt/gitlab/embedded/lib/ruby/site_ruby/2.7.0/bundler/cli/exec.rb:28:in `run'
/opt/gitlab/embedded/lib/ruby/site_ruby/2.7.0/bundler/cli.rb:476:in `exec'
/opt/gitlab/embedded/lib/ruby/site_ruby/2.7.0/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
/opt/gitlab/embedded/lib/ruby/site_ruby/2.7.0/bundler/vendor/thor/lib/thor/invocation.rb:127:in `invoke_command'
/opt/gitlab/embedded/lib/ruby/site_ruby/2.7.0/bundler/vendor/thor/lib/thor.rb:399:in `dispatch'
/opt/gitlab/embedded/lib/ruby/site_ruby/2.7.0/bundler/cli.rb:30:in `dispatch'
/opt/gitlab/embedded/lib/ruby/site_ruby/2.7.0/bundler/vendor/thor/lib/thor/base.rb:476:in `start'
/opt/gitlab/embedded/lib/ruby/site_ruby/2.7.0/bundler/cli.rb:24:in `start'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/bundler-2.1.4/exe/bundle:46:in `block in <top (required)>'
/opt/gitlab/embedded/lib/ruby/site_ruby/2.7.0/bundler/friendly_errors.rb:123:in `with_friendly_errors'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/bundler-2.1.4/exe/bundle:34:in `<top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'

The Log during upgrade

The log of postgresql:

2021-08-11_09:57:51.11324 LOG:  lock file "postmaster.pid" contains wrong PID: 563 instead of 309
2021-08-11_09:57:51.11660 LOG:  performing immediate shutdown because data directory lock file is invalid
2021-08-11_09:57:51.14100 LOG:  received immediate shutdown request
2021-08-11_09:57:51.14500 WARNING:  terminating connection because of crash of another server process
2021-08-11_09:57:51.15688 DETAIL:  The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
....
2021-08-11_10:02:56.57011 LOG:  could not open file "./server.crt": Operation not permitted
2021-08-11_10:02:56.57186 LOG:  could not open file "./server.key": Operation not permitted
2021-08-11_10:02:56.59008 LOG:  invalid resource manager ID in primary checkpoint record
2021-08-11_10:02:56.59256 PANIC:  could not locate a valid checkpoint record
2021-08-11_10:02:56.59356 LOG:  startup process (PID 5351) was terminated by signal 6: Aborted
2021-08-11_10:02:56.59492 LOG:  aborting startup due to startup process failure
2021-08-11_10:02:56.60138 LOG:  database system is shut down
2021-08-11_10:02:56.66310 LOG:  starting PostgreSQL 12.5 on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0, 64-bit
2021-08-11_10:02:56.67783 LOG:  listening on Unix socket "/var/opt/gitlab/postgresql/.s.PGSQL.5432"
2021-08-11_10:02:56.73906 LOG:  database system was interrupted; last known up at 2021-08-11 10:01:53 GMT

application.log

2021-08-11T09:57:53.260Z: Cached record for ApplicationSetting couldn't be loaded, falling back to uncached record: FATAL:  the database system is starting up

2021-08-11T09:59:00.366Z: Bypassing session in admin mode for: 10
2021-08-11T09:59:05.258Z: ActiveRecord connection established
2021-08-11T09:59:09.396Z: Checking state of dynamic postgres partitions
2021-08-11T09:59:09.411Z: Failed to create partition(s) for audit_events: PG::ConnectionBad: FATAL:  the database system is starting up

2021-08-11T09:59:09.428Z: Failed to create partition(s) for web_hook_logs: PG::ConnectionBad: FATAL:  the database system is starting up

2021-08-11T09:59:13.508Z: Cached record for ApplicationSetting couldn't be loaded, falling back to uncached record: FATAL:  the database system is starting up

2021-08-11T09:59:26.680Z: Cached record for ApplicationSetting couldn't be loaded, falling back to uncached record: could not connect to server: Connection refused
	Is the server running locally and accepting
	connections on Unix domain socket "/var/opt/gitlab/postgresql/.s.PGSQL.5432"?

...

2021-08-11T10:00:15.442Z: Cached record for ApplicationSetting couldn't be loaded, falling back to uncached record: FATAL:  the database system is starting up

2021-08-11T10:00:16.094Z: StuckCiJobsWorker: Cleaning stuck builds
2021-08-11T10:00:18.461Z: Cached record for ApplicationSetting couldn't be loaded, falling back to uncached record: FATAL:  the database system is starting up

2021-08-11T10:00:47.487Z: Bypassing session in admin mode for: 10
2021-08-11T10:00:54.993Z: ActiveRecord connection established
2021-08-11T10:00:59.021Z: Checking state of dynamic postgres partitions
2021-08-11T10:00:59.223Z: {:method=>"with_lock_retries", :class=>"Gitlab::Database::Partitioning::PartitionCreator", :message=>"Lock timeout is set", :current_iteration=>1, :lock_timeout_in_ms=>100}
2021-08-11T10:00:59.402Z: Failed to create partition(s) for web_hook_logs: ActiveRecord::StatementInvalid: PG::InvalidObjectDefinition: ERROR:  "web_hook_logs" is not partitioned

gitlab-rails-db-migrate-2021-08-11-10-00-02.log.1.gz:

== 20201218194311 CreateAdminNotes: migrating =================================
-- create_table(:namespace_admin_notes, {})
-- quote_column_name(:note)
   -> 0.0001s
   -> 0.5154s
-- quote_table_name("check_e9d2e71b5d")
   -> 0.0001s
-- quote_table_name(:namespace_admin_notes)
   -> 0.0000s
-- execute("ALTER TABLE \"namespace_admin_notes\"\nADD CONSTRAINT \"check_e9d2e71b5d\" CHECK (char_length(\"note\") <= 1000)\n")
   -> 0.0927s
== 20201218194311 CreateAdminNotes: migrated (0.6250s) ========================

== 20210106061254 AddUniqueIndexForGolangPackages: migrating ==================
-- transaction_open?()
   -> 0.0001s
-- index_exists?(:packages_packages, [:project_id, :name, :version], {:unique=>true, :where=>"package_type = 8", :name=>"index_packages_on_project_id_name_version_unique_when_golang", :algorithm=>:concurrently})
   -> 0.1665s
-- execute("SET statement_timeout TO 0")
   -> 0.0009s
-- add_index(:packages_packages, [:project_id, :name, :version], {:unique=>true, :where=>"package_type = 8", :name=>"index_packages_on_project_id_name_version_unique_when_golang", :algorithm=>:concurrently})
   -> 10.7077s
-- execute("RESET ALL")
   -> 0.0017s
...
== 20210311120154 InitializeConversionOfPushEventPayloadsEventIdToBigint: migrating 
-- table_exists?(:push_event_payloads)
   -> 0.0188s
-- column_exists?(:push_event_payloads, :event_id)
   -> 0.0100s
-- column_exists?(:push_event_payloads, :event_id)
   -> 0.0030s
-- columns(:push_event_payloads)
   -> 0.0052s
-- add_column(:push_event_payloads, "event_id_convert_to_bigint", :bigint, {:default=>0, :null=>false})NOTICE:  trigger "trigger_07c94931164e" for relation "push_event_payloads" does not exist, skipping
WARNING:  terminating connection because of crash of another server process
DETAIL:  The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
HINT:  In a moment you should be able to reconnect to the database and repeat your command.
rake aborted!
ActiveRecord::StatementInvalid: PG::ConnectionBad: PQconsumeInput() server closed the connection unexpectedly
	This probably means the server terminated abnormally
	before or while processing the request.
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:61:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'

Caused by:
PG::ConnectionBad: PQconsumeInput() server closed the connection unexpectedly
	This probably means the server terminated abnormally
	before or while processing the request.
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:61:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'

Caused by:
StandardError: An error has occurred, all later migrations canceled:

PG::UnableToSend: no connection to the server
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:341:in `ensure in disable_statement_timeout'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:341:in `disable_statement_timeout'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:152:in `add_concurrent_index'
/opt/gitlab/embedded/service/gitlab-rails/db/migrate/20210312140029_add_owner_and_id_index_on_active_ci_pipeline_schedules.rb:12:in `up'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:61:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'

Caused by:
ActiveRecord::StatementInvalid: PG::UnableToSend: no connection to the server
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:341:in `ensure in disable_statement_timeout'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:341:in `disable_statement_timeout'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:152:in `add_concurrent_index'
/opt/gitlab/embedded/service/gitlab-rails/db/migrate/20210312140029_add_owner_and_id_index_on_active_ci_pipeline_schedules.rb:12:in `up'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:61:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'

Caused by:
PG::UnableToSend: no connection to the server
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:341:in `ensure in disable_statement_timeout'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:341:in `disable_statement_timeout'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:152:in `add_concurrent_index'
/opt/gitlab/embedded/service/gitlab-rails/db/migrate/20210312140029_add_owner_and_id_index_on_active_ci_pipeline_schedules.rb:12:in `up'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:61:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'

Caused by:
ActiveRecord::StatementInvalid: PG::ConnectionBad: PQconsumeInput() server closed the connection unexpectedly
	This probably means the server terminated abnormally
	before or while processing the request.
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:153:in `block in add_concurrent_index'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:339:in `disable_statement_timeout'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:152:in `add_concurrent_index'
/opt/gitlab/embedded/service/gitlab-rails/db/migrate/20210312140029_add_owner_and_id_index_on_active_ci_pipeline_schedules.rb:12:in `up'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:61:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'

Caused by:
PG::ConnectionBad: PQconsumeInput() server closed the connection unexpectedly
	This probably means the server terminated abnormally
	before or while processing the request.
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:153:in `block in add_concurrent_index'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:339:in `disable_statement_timeout'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:152:in `add_concurrent_index'
/opt/gitlab/embedded/service/gitlab-rails/db/migrate/20210312140029_add_owner_and_id_index_on_active_ci_pipeline_schedules.rb:12:in `up'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:61:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Tasks: TOP => db:migrate
(See full trace by running task with --trace)

   -> 0.0115s
== 20210311120154 InitializeConversionOfPushEventPayloadsEventIdToBigint: migrated (0.0755s) 

== 20210311120155 BackfillEventsIdForBigintConversion: migrating ==============
== 20210311120155 BackfillEventsIdForBigintConversion: migrated (0.0001s) =====

== 20210311120156 BackfillPushEventPayloadEventIdForBigintConversion: migrating 
== 20210311120156 BackfillPushEventPayloadEventIdForBigintConversion: migrated (0.0001s) 

== 20210312140029 AddOwnerAndIdIndexOnActiveCiPipelineSchedules: migrating ====
-- transaction_open?()
   -> 0.0000s
-- index_exists?(:ci_pipeline_schedules, [:owner_id, :id], {:where=>"active = TRUE", :name=>"index_ci_pipeline_schedules_on_owner_id_and_id_and_active", :algorithm=>:concurrently})
   -> 0.0798s
-- execute("SET statement_timeout TO 0")
   -> 0.0012s
-- add_index(:ci_pipeline_schedules, [:owner_id, :id], {:where=>"active = TRUE", :name=>"index_ci_pipeline_schedules_on_owner_id_and_id_and_active", :algorithm=>:concurrently})
-- execute("RESET ALL")