Gitlab CE all pages load slowly

Gitlab is decidedly not fast and snappy on my server. Opening the main page in an already-logged-in state takes about five seconds to load, and I am on the same gigabit LAN as the server. Clicking around the UI is similarly slow. It looks just like problems I have seen with other software where reverse DNS is not available and the program hangs until the DNS request times out. In my case, I have reverse DNS entries for ALL the private IPs on my network, and all the names used in reverse dns have A records that match, so I know that is not the problem. Below are some DNS lookups on the server that is running gitlab. These take almost no time to return:

elyograg@smeagol:~$ host 192.168.217.222
222.217.168.192.in-addr.arpa domain name pointer 222.217.int.elyograg.org.
elyograg@smeagol:~$ host 222.217.int.elyograg.org.
222.217.int.elyograg.org has address 192.168.217.222
elyograg@smeagol:~$ host 192.168.217.200
200.217.168.192.in-addr.arpa domain name pointer smeagol.elyograg.org.
elyograg@smeagol:~$ host smeagol.elyograg.org.
smeagol.elyograg.org has address 192.168.217.200

Here are my gitlab versions:

elyograg@smeagol:~$ dpkg -l | grep gitlab
ii  gitlab-ce                                  15.4.0-ce.0                                  amd64        GitLab Community Edition (including NGINX, Postgres, Redis)
ii  gitlab-runner                              15.4.0                                       amd64        GitLab Runner

==========================

Detailed server specs:

Dell PowerEdge R720xd

Two 6-core 2Ghz Intel E5-2620 CPUs.
64GB RAM in two 4-slot banks of 8GB ECC.

Root volume is a pair of mirrored 300GB 2.5 inch 10000 RPM SAS drives.
Gitlab resides entirely on the root volume.
There is a big RAID10 volume of 12 4TB drives as well, gitlab backs up to that volume.
The RAID controller has 1GB of battery-backed cache memory.

This is NOT a wimpy VM. It’s high end bare metal. It should have more than enough capacity to run gitlab VERY quickly.

Running the ombnibus CE version on Ubuntu 22.04. I went with CE because most of the time I can support myself and don’t really want to cover a subscription for something I only need once in a while.

I am running gitlab behind haproxy. The proxy provides TLS and a few other things. The other things are not configured for the gitlab hostname. The proxy is speaking to the omnibus nginx server (running on the same box) without encryption. All my websites are behind haproxy, and other websites that I am running are VERY fast.

Is there possibly some configuration that is not set optimally? Here is the gitlab.rb file with comments and blank lines removed, and secrets redacted:

https://paste.elyograg.org/view/ad2f61aa

Here is the last few lines from “du -xh /var/opt/gitlab | sort -h”:

1.9G	/var/opt/gitlab/git-data
1.9G	/var/opt/gitlab/git-data/repositories
1.9G	/var/opt/gitlab/git-data/repositories/@hashed
1.9G	/var/opt/gitlab/prometheus
1.9G	/var/opt/gitlab/prometheus/data
4.4G	/var/opt/gitlab

If there is other information I can provide, let me know. If you want to see it in action for yourself, signups are enabled on the instance, though I do have it set to require admin approval. The URL is in the gitlab.rb paste.

While gathering information for this topic, I made a discovery. I found that once I had visited a particular page in the gitlab UI, that going back to it was faster. This time taking 2 seconds or less to load instead of 5 or more seconds. My guess is that the page was cached, probably by redis. This is really good, but the fact that it takes 5 or more seconds to load an uncached page is still a problem. With the server specs and the small amount of data being handled by my instance, I would expect to see the uncached page load to take 2 seconds or less and the cached page load to be pretty much instantaneous.

Also, it takes nearly two minutes for the server to get back to a ready state after a restart or a reconfigure. That seems excessive to me with my server specs.

I posted this in General because the other categories did not seem to apply.

If I restart or reconfigure, then everything is back to being slow, until it manages to cache some data.