Bad LFS performance


#1

We have a repository that use LFS, and fetching the objects managed by LFS is becoming slower and slower. Switching to a branch downloads the LFS objects, and while I have 73 MBps of bandwidth between my computer (MacBook Pro mid-2014) and our GitLab CE server, it takes 2 minutes and 44 seconds to download 564.65 MBps, while it should have taken 8 seconds. Copying a file of that size by SCP takes 10 seconds, so it seems it’s the LFS stack who is slow.

Server is Ubuntu 14.04 LTS, with 2 CPU cores, 8 GB of RAM and SSD storage. GitLab 9.1.1 Omnibus. 16 Unicorn workers.


#2

We also get 500 Internal Error problems when LFS is fetching:

2017/05/18 12:11:31 [error] 20201#0: *5329988 upstream prematurely closed connection while reading upstream, client: 192.168.1.213, server: xxxx.xx, request: “POST /dev/xxxxx.git/info/lfs/objects/batch HTTP/1.1”, upstream: “http://unix:/var/opt/gitlab/gitlab-workhorse/socket:/dev/xxxxx.git git/info/lfs/objects/batch”, host: “xxxx.xx”

Timeout between Nginx and the backend?