I posted this initially to Reddit here: Reddit - Dive into anything
We did a ton of troubleshooting there, all in the comments of that post. While everything seemed to turn up good with respect to hardware and my stack, I still can’t get things like page load times down below 2-3 seconds at the bare minimum. Usually requests take 7-10 seconds. It’s all over the place. I’ll paste the body of that post below for convenience :
Hey all,
Like the title says - I’m self hosting now version 16.1.2, the lastest, and page loads on average (according to the performance bar) take like 7 - 10+ seconds, even on subsequent reloads where the pages should be cached. Nothing really seems out of spec - database timings seem normalish, Redis timings seem good, but the request times are absolutely abysmal. I have no idea how to read the wall/cpu/object graphs.
The environment I’m hosting this in should be more than sufficient:
- 16 CPU cores, 3GHz
- 32GB DDR4 RAM
- SSD drives
I keep provisioning more and more resources to the Gitlab VM, but it doesn’t seem to make any difference. I used to run it in a ~2.1GHz environment, upgraded to the 3GHz and saw nearly no improvement.
I’ve set puma['worker_processes'] = 16 to match the CPU core count, nothing. I currently only have three users on this server, but I can’t really see adding more with how slow everything is to load. Am I missing something? How can I debug this?
Hi,
I’d be tempted to do something like this to get back to basics:
- Make a backup of
/etc/gitlab/gitlab.rb
- Copy
/opt/gitlab/etc/gitlab.rb.template to /etc/gitlab/gitlab.rb
- Make absolute minimal changes to
/etc/gitlab/gitlab.rb like external_url, plus any other basic stuff you need like email config, storage location if it was changed, or LDAP config if not using internal authentication, gitlab registry if you are using it for container images.
- Run
gitlab-ctl reconfigure and then gitlab-ctl restart.
You may need some additional stuff from your old gitlab.rb if something else is amiss, but apply stuff for functionality, rather than any tuning changes you’ve made. And then see how Gitlab performs.
It could well be during all the tweaking something has been done that just doesn’t sit right hence the suggestion to get back to a basic config as possible and then go from there.
Another option is make a backup of your gitlab, and then with the backup and /etc/gitlab/gitlab.rb (original one) and gitlab-secrets.json - restore this to a VM completely outside of your environment, eg: spin up a VM on DO, Linode, OVH or wherever just for a day or so. Restore your Gitlab and see how it performs there with the same configuration as your install. (unless you already did this).
I use a VM with 4vcpu, 24gb ram, 100gb at OVH and get 2 second page loads pretty much everywhere. Ram usage is about 7GB with pretty much the default basic gitlab config (no tuning except disabling non-needed gitlab services like KAS, etc).