We have a repository that use LFS, and fetching the objects managed by LFS is becoming slower and slower. Switching to a branch downloads the LFS objects, and while I have 73 MBps of bandwidth between my computer (MacBook Pro mid-2014) and our GitLab CE server, it takes 2 minutes and 44 seconds to download 564.65 MBps, while it should have taken 8 seconds. Copying a file of that size by SCP takes 10 seconds, so it seems it’s the LFS stack who is slow.
Server is Ubuntu 14.04 LTS, with 2 CPU cores, 8 GB of RAM and SSD storage. GitLab 9.1.1 Omnibus. 16 Unicorn workers.
We see the same thing on our end: pulling LFS objects seems to have a slower transfer rate than SCP or even vanilla git pull. I wonder what’s going on here…
Feels like we should offload our LFS storage to an S3 bucket or something, but honestly if benchmarking against SCP/Git-proper within a same-machine setup shows this big a difference, it doesn’t seem likely to be “The Big Fix” we’re looking for.
Unless I’m misunderstanding how the default configuration for the built-in Git LFS server works?
P.S. we’re running a self-managed instance of GitLab EE 15.x.
Seeing the same with current latest gitlab-ce, took 50 minutes to upload LFS-heavy repo (initial push) of 2.6gb total, despite my home connection is a stable 400mbps upload optics, and the server was not overloaded (cpu, memory, disk io, was all at normal utilization rates in aws ec2).
$ time git push --set-upstream origin master
info: detecting host provider for 'https://xxxxxxxxxxxxx/'...
info: detecting host provider for 'https://xxxxxxxxxxxxx/'...
Locking support detected on remote "origin". Consider enabling it with:
$ git config lfs.https://xxxxxxxxxx/root/speed-test.git/info/lfs.locksverify true
Uploading LFS objects: 100% (6331/6331), 2.6 GB | 1.1 MB/s, done.
Enumerating objects: 96747, done.
Counting objects: 100% (96747/96747), done.
Delta compression using up to 16 threads
Compressing objects: 100% (67963/67963), done.
Writing objects: 100% (96746/96746), 362.91 MiB | 763.00 KiB/s, done.
Total 96746 (delta 23297), reused 96746 (delta 23297), pack-reused 0
remote: Resolving deltas: 100% (23297/23297), done.
To ssh://xxxxxxxxxxxxxx/root/speed-test.git
3375f500e..2fb2fb0e0 master -> master
Branch 'master' set up to track remote branch 'master' from 'origin'.
real 51m48.787s
user 0m0.015s
sys 0m0.031s
were you guys able to find solutions to the old problem? I know it’s reviving a very old thread, but seems like LFS performance issues are still here in 2023
One theory I have is that authentication is used for PUT requests as shown in trace mode, but even that can’t explain the 50 minutes total time and average 1.1 MB/s , not sure what to think of it. Any ideas about how to turn off auth for LFS or to improve this? We’re planning to use gitlab as a local LFS storage server within the internal network only, between CI servers and Git servers, so the LFS box can be without auth at all, no external access to network is possible. But I didn’t find a setting to turn off the auth, looks like each PUT request will first grab an auth token and then upload a file, which is pretty bad for performance.
