Praefect and its dedicated postgresql does not work

hi all,
I’ve met a problem when I try to install a gitlab instance through omnibus.

the basic information is below:
1 node for redis and consul
1 node for postgresql (for gitlab)
1 praefect
1 praefect dedicated postgresql
3 gitaly
1 gitlab rails
1 prometheus

I just follow the guide here: https://docs.gitlab.com/13.11/ee/administration/reference_architectures/3k_users.htm

It seems all nodes and components goes well before it prompt “500 error” in dashboard after I input admin user and password.
Then I check the logs for praefect, there are many errors like below:

{"level":"error","msg":"Error updating node: pq: current transaction is aborted, commands ignored until end of transaction block","pid":2781,"praefectName":"iZj6c3t5dfgdvx0duh5nhfZ:0.0.0.0:2305","time":"2021-04-26T17:24:17.357Z","virtual_storage":"default"}
{"address":"tcp://172.25.94.51:8075","error":"pq: current transaction is aborted, commands ignored until end of transaction block","level":"error","msg":"error checking node","pid":2781,"praefectName":"iZj6c3t5dfgdvx0duh5nhfZ:0.0.0.0:2305","storage":"gitaly-2","time":"2021-04-26T17:24:17.357Z","virtual_storage":"default"}
{"error":"error retrieving quorum count: pq: current transaction is aborted, commands ignored until end of transaction block","level":"error","msg":"unable to validate primary","pid":2781,"praefectName":"iZj6c3t5dfgdvx0duh5nhfZ:0.0.0.0:2305","time":"2021-04-26T17:24:17.358Z","virtual_storage":"default"}
{"error":"pq: Could not complete operation in a failed transaction","level":"error","msg":"failed committing transaction","pid":2781,"praefectName":"iZj6c3t5dfgdvx0duh5nhfZ:0.0.0.0:2305","time":"2021-04-26T17:24:17.359Z","virtual_storage":"default"}
{"component":"replication_manager","error":"query: pq: relation \"replication_queue_lock\" does not exist","level":"error","msg":"failed to dequeue replication events","pid":2781,"replication_job_target":"gitaly-1","time":"2021-04-26T17:24:18.696Z","virtual_storage":"default"}
{"component":"replication_manager","error":"query: pq: relation \"replication_queue_lock\" does not exist","level":"error","msg":"failed to dequeue replication events","pid":2781,"replication_job_target":"gitaly-2","time":"2021-04-26T17:24:18.697Z","virtual_storage":"default"}
{"component":"replication_manager","error":"query: pq: relation \"replication_queue_lock\" does not exist","level":"error","msg":"failed to dequeue replication events","pid":2781,"replication_job_target":"gitaly-3","time":"2021-04-26T17:24:18.698Z","virtual_storage":"default"}
{"level":"error","msg":"Error updating node: pq: relation \"node_status\" does not exist","pid":2781,"praefectName":"iZj6c3t5dfgdvx0duh5nhfZ:0.0.0.0:2305","time":"2021-04-26T17:24:20.354Z","virtual_storage":"default"}
{"address":"tcp://172.25.94.49:8075","error":"pq: relation \"node_status\" does not exist","level":"error","msg":"error checking node","pid":2781,"praefectName":"iZj6c3t5dfgdvx0duh5nhfZ:0.0.0.0:2305","storage":"gitaly-1","time":"2021-04-26T17:24:20.354Z","virtual_storage":"default"}
{"level":"error","msg":"Error updating node: pq: current transaction is aborted, commands ignored until end of transaction block","pid":2781,"praefectName":"iZj6c3t5dfgdvx0duh5nhfZ:0.0.0.0:2305","time":"2021-04-26T17:24:20.355Z","virtual_storage":"default"}
{"address":"tcp://172.25.94.54:8075","error":"pq: current transaction is aborted, commands ignored until end of transaction block","level":"error","msg":"error checking node","pid":2781,"praefectName":"iZj6c3t5dfgdvx0duh5nhfZ:0.0.0.0:2305","storage":"gitaly-3","time":"2021-04-26T17:24:20.355Z","virtual_storage":"default"}
{"level":"error","msg":"Error updating node: pq: current transaction is aborted, commands ignored until end of transaction block","pid":2781,"praefectName":"iZj6c3t5dfgdvx0duh5nhfZ:0.0.0.0:2305","time":"2021-04-26T17:24:20.356Z","virtual_storage":"default"}
{"address":"tcp://172.25.94.51:8075","error":"pq: current transaction is aborted, commands ignored until end of transaction block","level":"error","msg":"error checking node","pid":2781,"praefectName":"iZj6c3t5dfgdvx0duh5nhfZ:0.0.0.0:2305","storage":"gitaly-2","time":"2021-04-26T17:24:20.356Z","virtual_storage":"default"}
{"error":"error retrieving quorum count: pq: current transaction is aborted, commands ignored until end of transaction block","level":"error","msg":"unable to validate primary","pid":2781,"praefectName":"iZj6c3t5dfgdvx0duh5nhfZ:0.0.0.0:2305","time":"2021-04-26T17:24:20.356Z","virtual_storage":"default"}
{"error":"pq: Could not complete operation in a failed transaction","level":"error","msg":"failed committing transaction","pid":2781,"praefectName":"iZj6c3t5dfgdvx0duh5nhfZ:0.0.0.0:2305","time":"2021-04-26T17:24:20.357Z","virtual_storage":"default"}

and there are no table and any data in praefect’s postgresql database:

[root@iZj6c5pww19e8nixrn9wccZ ~]# /opt/gitlab/embedded/bin/psql -U praefect -d praefect_production -h 172.30.23.190
psql (12.6)
SSL connection (protocol: TLSv1.3, cipher: TLS_AES_256_GCM_SHA384, bits: 256, compression: off)
Type "help" for help.

praefect_production=> \l
                                             List of databases
        Name         |    Owner    | Encoding |   Collate   |    Ctype    |        Access privileges
---------------------+-------------+----------+-------------+-------------+---------------------------------
 gitlabhq_production | gitlab      | UTF8     | en_US.UTF-8 | en_US.UTF-8 |
 postgres            | gitlab-psql | UTF8     | en_US.UTF-8 | en_US.UTF-8 |
 praefect_production | praefect    | UTF8     | en_US.UTF-8 | en_US.UTF-8 |
 template0           | gitlab-psql | UTF8     | en_US.UTF-8 | en_US.UTF-8 | =c/"gitlab-psql"               +
                     |             |          |             |             | "gitlab-psql"=CTc/"gitlab-psql"
 template1           | gitlab-psql | UTF8     | en_US.UTF-8 | en_US.UTF-8 | =c/"gitlab-psql"               +
                     |             |          |             |             | "gitlab-psql"=CTc/"gitlab-psql"
(5 rows)

praefect_production=> \dt
Did not find any relations.
praefect_production=>

It seems praefect’s database instance (praefect_production) is not initialized at all!
I am not sure when&where&who takes reponsible for this job? “gitlab-rake gitlab:db:configure” command? what should I do to fix this issue?

Thanks very much!

BTW, these are some configuration message of my gitlab instance:

version: gitlab-ee 13.10

praefect configuration:

postgresql['enable'] = false
redis['enable'] = false
gitaly['enable'] = false
nginx['enable'] = false
puma['enable'] = false
unicorn['enable'] = false
sidekiq['enable'] = false
gitlab_workhorse['enable'] = false
grafana['enable'] = false

# If you run a separate monitoring node you can disable these services
alertmanager['enable'] = false
prometheus['enable'] = false

# Praefect Configuration
praefect['enable'] = true
praefect['listen_addr'] = '0.0.0.0:2305'

# Prevent database migrations from running on upgrade automatically
praefect['auto_migrate'] = false
gitlab_rails['auto_migrate'] = false
gitlab_rails['rake_cache_clear'] = false

# Praefect External Token
# This is needed by clients outside the cluster (like GitLab Shell) to communicate with the Praefect cluster
praefect['auth_token'] = 'praefect_external_token'

# Praefect Database Settings
praefect['database_host'] = '172.30.23.190'
praefect['database_port'] = 5432
# `no_proxy` settings must always be a direct connection for caching
praefect['database_host_no_proxy'] = '172.30.23.190'
praefect['database_port_no_proxy'] = 5432
praefect['database_dbname'] = 'praefect_production'
praefect['database_user'] = 'praefect'
praefect['database_password'] = 'praefect'

# Praefect Virtual Storage config
# Name of storage hash must match storage name in git_data_dirs on GitLab
# server ('praefect') and in git_data_dirs on Gitaly nodes ('gitaly-1')
praefect['virtual_storages'] = {
  'default' => {
    'gitaly-1' => {
      'address' => 'tcp://172.25.94.49:8075',
      'token'   => 'praefect_internal_token',
      'primary' => 'true'
    },
    'gitaly-2' => {
      'address' => 'tcp://172.25.94.51:8075',
      'token'   => 'praefect_internal_token'
    },
    'gitaly-3' => {
      'address' => 'tcp://172.25.94.54:8075',
      'token'   => 'praefect_internal_token'
    },
  }
}

# Set the network addresses that the exporters will listen on for monitoring
node_exporter['listen_address'] = '0.0.0.0:9100'
praefect['prometheus_listen_addr'] = '0.0.0.0:9652'

## The IPs of the Consul server nodes
# Configure the Consul agent
consul['enable'] = true
consul['monitoring_service_discovery'] =  true
consul['configuration'] = {
   bind_addr: '172.25.94.47',
   retry_join: %w(172.25.94.48),
}

gitaly configuration:

postgresql['enable'] = false
redis['enable'] = false
nginx['enable'] = false
puma['enable'] = false
unicorn['enable'] = false
sidekiq['enable'] = false
gitlab_workhorse['enable'] = false
grafana['enable'] = false

# If you run a separate monitoring node you can disable these services
alertmanager['enable'] = false
prometheus['enable'] = false

# Prevent database migrations from running on upgrade automatically
gitlab_rails['auto_migrate'] = false

# Configure the gitlab-shell API callback URL. Without this, `git push` will
# fail. This can be your 'front door' GitLab URL or an internal load balancer.
# Don't forget to copy `/etc/gitlab/gitlab-secrets.json` from web server to Gitaly server.
gitlab_rails['internal_api_url'] = "https://test.jihulab.com"

# Make Gitaly accept connections on all network interfaces. You must use firewalls to restrict access to this address/port.
# Comment out following line if you only want to support TLS connections
gitaly['listen_addr'] = "0.0.0.0:8075"

# Gitaly Auth Token
# Should be the same as praefect_internal_token
gitaly['auth_token'] = "praefect_internal_token"

git_data_dirs({
  "gitaly-1" => {
    "path" => "/var/opt/gitlab/git-data"
   },
   "gitaly-2" => {
    "path" => "/var/opt/gitlab/git-data"
   },
  "gitaly-3" => {
    "path" => "/var/opt/gitlab/git-data"
  }
})

gitlab-rails configuration:

external_url 'https://test.jihulab.com'

git_data_dirs({
  "default" => {
    "gitaly_address" => "tcp://172.25.94.47:2305",
    "gitaly_token" => 'praefect_external_token'
  }
})
roles ['application_role']
gitaly['enable'] = false
nginx['enable'] = true
sidekiq['enable'] = false

postgresql['enable'] = false
gitlab_rails['db_host'] = '172.30.23.186'
gitlab_rails['db_port'] = 5432
gitlab_rails['db_password'] = 'gitlab'
# Prevent database migrations from running on upgrade automatically
gitlab_rails['auto_migrate'] = false

## The same password for Redis authentication you set up for the Redis primary node.
redis['master_password'] = 'redis-password-goes-here'

## A list of sentinels with `host` and `port`
gitlab_rails['redis_sentinels'] = [
  {'host' => '172.25.94.48', 'port' => 26379}
]

## Enable service discovery for Prometheus
consul['enable'] = true
consul['monitoring_service_discovery'] =  true
consul['configuration'] = {
   bind_addr: '172.30.23.188',
   retry_join: %w(172.25.94.48),
}

# Set the network addresses that the exporters used for monitoring will listen on
node_exporter['listen_address'] = '0.0.0.0:9100'
gitlab_workhorse['prometheus_listen_addr'] = '0.0.0.0:9229'
sidekiq['listen_address'] = "0.0.0.0"
puma['listen'] = '0.0.0.0'

# Add the monitoring node's IP address to the monitoring whitelist and allow it to scrape the NGINX metrics
gitlab_rails['monitoring_whitelist'] = ['172.30.23.185/32', '172.0.0.0/8']
nginx['status']['options']['allow'] = ['172.30.23.185/32', '172.0.0.0/8']
gitlab_rails['prometheus_address'] = '172.30.23.185:9090'

praefect be able to reach postgresql successfully with below test command:
sudo -u git /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml sql-ping

praefect to gitaly is also ok:
sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml dial-nodes

gitlab-rake gitlab:db:configure command will not initialized praefect’s dedicated postgresql