We are running a self-hosted GitLab instance, and sometimes (several times a week) it becomes totally unresponsive. Evey page load takes tens of seconds. Yesterday this problem became so bad, that several teams are totally stuck.
I have a Grafana screenshot demonstrating this problem.
Note that the latency rises when the workhorse's 3xx codes change to 2xx codes. The “duration_ms” of these 2xx responses is ~35000 according to logs.
We rebooted the server in ~14:10. After this, the problem went away, but not for long.
Today the problem appeared early morning and paused the work…
We tried to switch from puma to unicorn, it does not help…
Does anybody have any idea what to check?
In fact, we have some huge merge requests, but I guess they should not slow down the entire instance…