TL;DR plain vanilla gitlab-ce ubuntu instance web interface very slow until reboot. Why? Suggestions?
I’m running a local gitlab-ce instance. It’s on Ubuntu 16.04LTE and I keep it up to date with apt. It’s on a 4GB vm with two cores x two chips, on a Xen hypervisor on an HP G7 in the server closet.
It has about five users.
It’s simple. We don’t have any CI runners yet. We don’t have any submodule dependencies. About ten repos. http access only, it’s within our firewall. We run a daily backup using gitlab-rake gitlab:backup:create in a cronjob. Plain vanilla setup, I think.
The web interface is very slow. When one user sends a hyperlink to a source line or wiki page to another user, the receiving user often waits almost a minute, and sometimes gets the ngnix 60-sec timeout.
The ssh interface for pulls / pushes doesn’t seem to be as slow. (That’s good).
Bouncing the server VM temporarily solves the problem.
Load averages and RAM are within reason. The disk volumes are half-full. There’s nothing obvious (to me) in the /var/log/… files, except this sequence in the redis log, once a second 24x7, even when nobody’s using the instance.
2016-12-13_11:53:39.06481 1061:M 13 Dec 06:53:39.064 * 10000 changes in 60 seconds. Saving...
2016-12-13_11:53:39.06593 1061:M 13 Dec 06:53:39.065 * Background saving started by pid 5560
2016-12-13_11:53:39.10352 5560:C 13 Dec 06:53:39.103 * DB saved on disk
2016-12-13_11:53:39.10414 5560:C 13 Dec 06:53:39.104 * RDB: 4 MB of memory used by copy-on-write
2016-12-13_11:53:39.16613 1061:M 13 Dec 06:53:39.165 * Background saving terminated with success
A couple of repos have outbound webhooks to Slack announcing commits. But this slowness seems to occur independently of commits.
I’m hoping for suggestions / advice on troubleshooting, and hopefully fixing, this performance problem. It would be nice to fix it before adding CI runners.