Huge performance problem


We are running GitLab 9.1.1 CE, and beginning this week, we have huge performance problem when pushing or pulling. I have a repository that usually takes a couple of seconds to update, and now it takes more than 5 minutes:

$ GIT_TRACE_PERFORMANCE=1 git push origin master
08:14:27.685191 trace.c:420             performance: 0.024578000 s: git command: 'git' 'credential-osxkeychain' 'get'
08:14:30.803367 trace.c:420             performance: 0.060249000 s: git command: 'git' 'credential-osxkeychain' 'store'
Counting objects: 4, done.
Writing objects: 100% (4/4), 177.11 KiB | 0 bytes/s, done.
Total 4 (delta 0), reused 0 (delta 0)
08:14:30.868995 trace.c:420             performance: 0.057733000 s: git command: 'git' 'pack-objects' '--all-progress-implied' '--revs' '--stdout' '--thin' '--delta-base-offset' '--progress'
remote: GitLab: Failed to authorize your Git request: internal API unreachable
08:19:32.638793 trace.c:420             performance: 301.830800000 s: git command: 'git' 'send-pack' '--stateless-rpc' '--helper-status' '--thin' '--progress' 'https://probert@git.druide/soutien_interne/configurations-puppet.git/' '--stdin'
To https://git.druide/soutien_interne/configurations-puppet.git
! [remote rejected] master -> master (pre-receive hook declined)
error: failed to push some refs to 'https://probert@git.druide/soutien_interne/configurations-puppet.git'
08:19:32.642790 trace.c:420             performance: 306.585836000 s: git command: '/Applications/' 'push' 'origin' 'master'

Running tcpdump, I don’t see any packets coming from or to the server. Same problem with both SSH or HTTPS. But we can access the Web interface without any performance problems, and copying a 1.5 GB file on the server is fast. iostat don’t show any problems, CPU is at 30% max, no swapping, and ifconfig doesn’t show any errors.

Server is running Ubuntu 14.04 LTS, two virtual CPUs, SSD storage.

Can not help really but we see a performance drop as well (actions which took 5 seconds with latest GL 8.x now take up to 40 seconds)

We had to raise the number of Unicorn workers to 16.

Same trouble…

We found out that the main problem was that the user who do Git operations on our build system is in LDAP, and GitLab was doing a lot (2 to 4 requests per second) of LDAP requests. We moved the user to a local one, and it solved a lot of problems.