hi all,
I’ve met a problem when I try to install a gitlab instance through omnibus.
the basic information is below:
1 node for redis and consul
1 node for postgresql (for gitlab)
1 praefect
1 praefect dedicated postgresql
3 gitaly
1 gitlab rails
1 prometheus
I just follow the guide here: https://docs.gitlab.com/13.11/ee/administration/reference_architectures/3k_users.htm
It seems all nodes and components goes well before it prompt “500 error” in dashboard after I input admin user and password.
Then I check the logs for praefect, there are many errors like below:
{"level":"error","msg":"Error updating node: pq: current transaction is aborted, commands ignored until end of transaction block","pid":2781,"praefectName":"iZj6c3t5dfgdvx0duh5nhfZ:0.0.0.0:2305","time":"2021-04-26T17:24:17.357Z","virtual_storage":"default"}
{"address":"tcp://172.25.94.51:8075","error":"pq: current transaction is aborted, commands ignored until end of transaction block","level":"error","msg":"error checking node","pid":2781,"praefectName":"iZj6c3t5dfgdvx0duh5nhfZ:0.0.0.0:2305","storage":"gitaly-2","time":"2021-04-26T17:24:17.357Z","virtual_storage":"default"}
{"error":"error retrieving quorum count: pq: current transaction is aborted, commands ignored until end of transaction block","level":"error","msg":"unable to validate primary","pid":2781,"praefectName":"iZj6c3t5dfgdvx0duh5nhfZ:0.0.0.0:2305","time":"2021-04-26T17:24:17.358Z","virtual_storage":"default"}
{"error":"pq: Could not complete operation in a failed transaction","level":"error","msg":"failed committing transaction","pid":2781,"praefectName":"iZj6c3t5dfgdvx0duh5nhfZ:0.0.0.0:2305","time":"2021-04-26T17:24:17.359Z","virtual_storage":"default"}
{"component":"replication_manager","error":"query: pq: relation \"replication_queue_lock\" does not exist","level":"error","msg":"failed to dequeue replication events","pid":2781,"replication_job_target":"gitaly-1","time":"2021-04-26T17:24:18.696Z","virtual_storage":"default"}
{"component":"replication_manager","error":"query: pq: relation \"replication_queue_lock\" does not exist","level":"error","msg":"failed to dequeue replication events","pid":2781,"replication_job_target":"gitaly-2","time":"2021-04-26T17:24:18.697Z","virtual_storage":"default"}
{"component":"replication_manager","error":"query: pq: relation \"replication_queue_lock\" does not exist","level":"error","msg":"failed to dequeue replication events","pid":2781,"replication_job_target":"gitaly-3","time":"2021-04-26T17:24:18.698Z","virtual_storage":"default"}
{"level":"error","msg":"Error updating node: pq: relation \"node_status\" does not exist","pid":2781,"praefectName":"iZj6c3t5dfgdvx0duh5nhfZ:0.0.0.0:2305","time":"2021-04-26T17:24:20.354Z","virtual_storage":"default"}
{"address":"tcp://172.25.94.49:8075","error":"pq: relation \"node_status\" does not exist","level":"error","msg":"error checking node","pid":2781,"praefectName":"iZj6c3t5dfgdvx0duh5nhfZ:0.0.0.0:2305","storage":"gitaly-1","time":"2021-04-26T17:24:20.354Z","virtual_storage":"default"}
{"level":"error","msg":"Error updating node: pq: current transaction is aborted, commands ignored until end of transaction block","pid":2781,"praefectName":"iZj6c3t5dfgdvx0duh5nhfZ:0.0.0.0:2305","time":"2021-04-26T17:24:20.355Z","virtual_storage":"default"}
{"address":"tcp://172.25.94.54:8075","error":"pq: current transaction is aborted, commands ignored until end of transaction block","level":"error","msg":"error checking node","pid":2781,"praefectName":"iZj6c3t5dfgdvx0duh5nhfZ:0.0.0.0:2305","storage":"gitaly-3","time":"2021-04-26T17:24:20.355Z","virtual_storage":"default"}
{"level":"error","msg":"Error updating node: pq: current transaction is aborted, commands ignored until end of transaction block","pid":2781,"praefectName":"iZj6c3t5dfgdvx0duh5nhfZ:0.0.0.0:2305","time":"2021-04-26T17:24:20.356Z","virtual_storage":"default"}
{"address":"tcp://172.25.94.51:8075","error":"pq: current transaction is aborted, commands ignored until end of transaction block","level":"error","msg":"error checking node","pid":2781,"praefectName":"iZj6c3t5dfgdvx0duh5nhfZ:0.0.0.0:2305","storage":"gitaly-2","time":"2021-04-26T17:24:20.356Z","virtual_storage":"default"}
{"error":"error retrieving quorum count: pq: current transaction is aborted, commands ignored until end of transaction block","level":"error","msg":"unable to validate primary","pid":2781,"praefectName":"iZj6c3t5dfgdvx0duh5nhfZ:0.0.0.0:2305","time":"2021-04-26T17:24:20.356Z","virtual_storage":"default"}
{"error":"pq: Could not complete operation in a failed transaction","level":"error","msg":"failed committing transaction","pid":2781,"praefectName":"iZj6c3t5dfgdvx0duh5nhfZ:0.0.0.0:2305","time":"2021-04-26T17:24:20.357Z","virtual_storage":"default"}
and there are no table and any data in praefect’s postgresql database:
[root@iZj6c5pww19e8nixrn9wccZ ~]# /opt/gitlab/embedded/bin/psql -U praefect -d praefect_production -h 172.30.23.190
psql (12.6)
SSL connection (protocol: TLSv1.3, cipher: TLS_AES_256_GCM_SHA384, bits: 256, compression: off)
Type "help" for help.
praefect_production=> \l
List of databases
Name | Owner | Encoding | Collate | Ctype | Access privileges
---------------------+-------------+----------+-------------+-------------+---------------------------------
gitlabhq_production | gitlab | UTF8 | en_US.UTF-8 | en_US.UTF-8 |
postgres | gitlab-psql | UTF8 | en_US.UTF-8 | en_US.UTF-8 |
praefect_production | praefect | UTF8 | en_US.UTF-8 | en_US.UTF-8 |
template0 | gitlab-psql | UTF8 | en_US.UTF-8 | en_US.UTF-8 | =c/"gitlab-psql" +
| | | | | "gitlab-psql"=CTc/"gitlab-psql"
template1 | gitlab-psql | UTF8 | en_US.UTF-8 | en_US.UTF-8 | =c/"gitlab-psql" +
| | | | | "gitlab-psql"=CTc/"gitlab-psql"
(5 rows)
praefect_production=> \dt
Did not find any relations.
praefect_production=>
It seems praefect’s database instance (praefect_production) is not initialized at all!
I am not sure when&where&who takes reponsible for this job? “gitlab-rake gitlab:db:configure” command? what should I do to fix this issue?
Thanks very much!