Gitlab is very slow to answer but no errors


I installed Gitlab (now version 7.12.2) omnibus on an Xeon X3450 2,67Ghz, 4G ram, ubuntu
,and many people report me that the access could be very slow (web interface but also git CLI). Slow means: the interface will takes 5-8sec to answer but I have seen 14sec for the webpage to load.
I read many bug reports but none of the solutions works for me. Gitlab does not see errors with self checks.
What is annoying is that some days everything is good (load with an average of 100ms) but the next day it will takes 10sec to load, but never fail to load.
This gitlab does not have big projects, there are few users (<30) so I don’t see any resons why it happen.

Do you have an idea ?
Thanks for any answer.