Git lfs authentication expire after 30 minutes, big file upload fails with 401 error

Hello. I’m trying to push some really big LFS files to locally hosted Gitlab CE server with git LFS support enabled. I’m using ssh protocol with public key stored on server for main git authentication. But it seems like git lfs uses it’s own authentication and it expires after 30 minutes. For example, if I try to push 18 Gb file over 100 Mbit LAN, it succeeds (upload of single file takes less than 30 minutes, while total time for git push takes more). If I try to push a 22 Gb file to the same repository, upload starts normally and then fails with authentication error. Upload of this single file takes more than 30 minutes (13:13-13:48 in the git trace log below). My best guess is that some part of authentication system expires after 30 minutes (of inactivity) and is not renewed. Maybe it’s a server configuration problem, but I can’t find any related setup option. Can you please check issue/add some hints about correct setup to make authentication token last longer?

Git lfs documentation suggests that that server can specify additional fields “expires_at”/“expires_in” in authentication details to provide lfs client with information about lifetime of authentication token.

Gitlab server version: 10.7 Community Edition

Client git lfs version:

$ git lfs version
git-lfs/2.4.0 (GitHub; linux amd64; go 1.8.3)

Authentication details:

$ ssh git@server git-lfs-authenticate user/git-lfs-test-3.git upload
{"header":{"Authorization":"Basic ZG1pdHJpeS56OlRxcDVmOHVueGRwenU4bVdEVW92Z0ZIR0N2Ml9HczVoQm9SN3VZZnlyQUV1NVRZZEN3"},"href":"https server address"}

Trace of failed push:

$ GIT_TRACE=1 git push origin master
13:13:02.749320 git.c:344               trace: built-in: git 'push' 'origin' 'master'
13:13:02.749939 run-command.c:334       trace: run_command: 'ssh' 'git@server' 'git-receive-pack '\''user/git-lfs-test-3.git'\'''
13:13:03.518181 run-command.c:334       trace: run_command: '.git/hooks/pre-push' 'origin' 'git@server:user/git-lfs-test-3.git'
13:13:03.583631 git.c:561               trace: exec: 'git-lfs' 'pre-push' 'origin' 'git@server:user/git-lfs-test-3.git'
13:13:03.583720 run-command.c:334       trace: run_command: 'git-lfs' 'pre-push' 'origin' 'git@server:user/git-lfs-test-3.git'
13:13:03.673086 trace git-lfs: exec: git 'version'
13:13:03.681341 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=cat' '-c' 'filter.lfs.clean=cat' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'rev-parse' 'HEAD' '--symbolic-full-name' 'HEAD'
13:13:03.687379 trace git-lfs: exec: git 'config' '-l'
13:13:03.689304 trace git-lfs: pre-push: refs/heads/master 709c0443eb59863fa545490f2999a4bdf2f9e288 refs/heads/master aa0e262f8ab2654e62eeb0bf78226bb3fc4e400e
13:13:04.284219 trace git-lfs: run_command: ssh -- git@server git-lfs-authenticate user/git-lfs-test-3.git upload
13:13:04.830866 trace git-lfs: HTTP: POST https :// server/user/git-lfs-test-3.git/info/lfs/locks/verify
13:13:05.014024 trace git-lfs: HTTP: 200
13:13:05.014223 trace git-lfs: HTTP: {"ours":[],"theirs":[]}
13:13:05.014400 trace git-lfs: tq: running as batched queue, batch size of 100
13:13:05.014765 trace git-lfs: run_command: git rev-list --stdin --objects --not --remotes=origin --
13:13:05.017356 trace git-lfs: run_command: git cat-file --batch
13:13:05.042195 trace git-lfs: tq: sending batch of size 1                                              
13:13:05.042565 trace git-lfs: ssh cache: git@server git-lfs-authenticate user/git-lfs-test-3.git upload
13:13:05.042902 trace git-lfs: api: batch 1 files
13:13:05.043149 trace git-lfs: HTTP: POST https :// server/user/git-lfs-test-3.git/info/lfs/objects/batch
13:13:05.578148 trace git-lfs: HTTP: 200
13:13:05.578314 trace git-lfs: HTTP: {"objects":[{"oid":"58051e32c37f78b536367ab1f0bd019259248bf144db92706f3c663c1378639f","size":23068672000,"actions":{"upload":{"href":"https :// server/user/git-lfs-test-3.git/gitlab-lfs/objects/58051e32c37f78b536367ab1f0bd019259248bf144db92706f3c663c1378639f/23068672000","header":{"Authorization":"Basic ZG1pdHJpeS56OnlaWWtmUjlhWFNGVWpvdUJyc3VrWnJzM182TW5jUXlhNDd5eTRMNUtfUzZnVnl4eC1n"}}}}]}
13:13:05.578635 trace git-lfs: tq: starting transfer adapter "basic"
13:13:05.579386 trace git-lfs: HTTP: PUT https :// server/user/git-lfs-test-3.git/gitlab-lfs/objects/58051e32c37f78b536367ab1f0bd019259248bf144db92706f3c663c1378639f/23068672000
13:48:40.086998 trace git-lfs: HTTP: 401 | 11 MB/s                                                      
13:48:40.087295 trace git-lfs: tq: retrying object 58051e32c37f78b536367ab1f0bd019259248bf144db92706f3c663c1378639f: Authentication required: Authorization error: https :// server/user/git-lfs-test-3.git/gitlab-lfs/objects/58051e32c37f78b536367ab1f0bd019259248bf144db92706f3c663c1378639f/23068672000
Check that you have proper access to the repository
13:48:40.087387 trace git-lfs: tq: enqueue retry #1 for "58051e32c37f78b536367ab1f0bd019259248bf144db92706f3c663c1378639f" (size: 23068672000)
13:48:40.087470 trace git-lfs: tq: sending batch of size 1
13:48:40.087765 trace git-lfs: ssh cache: git@server git-lfs-authenticate user/git-lfs-test-3.git upload
13:48:40.087830 trace git-lfs: api: batch 1 files
13:48:40.088030 trace git-lfs: HTTP: POST https :// server/user/git-lfs-test-3.git/info/lfs/objects/batch
13:48:40.885121 trace git-lfs: HTTP: 401
13:48:40.885230 trace git-lfs: api error: Authentication required: Authorization error: https :// server/user/git-lfs-test-3.git/info/lfs/objects/batch
Check that you have proper access to the repository
batch response: Authentication required: Authorization error: https :// server/user/git-lfs-test-3.git/info/lfs/objects/batch
Check that you have proper access to the repository
error: failed to push some refs to 'git@server:user/git-lfs-test-3.git'

Unfortunately, it looks like the timeout is hardcoded in the gitlab database: https://gitlab.com/gitlab-org/gitlab-ce/blob/master/lib/gitlab/lfs_token.rb (see EXPIRY_TIME)

You can extend the lifespan of the token indefinitely by running:

while sleep 120; do ssh git@server git-lfs-authenticate namespace/project.git upload; done

but be careful to not upload files which are too large for gitlab to handle