Git lfs batch intermittently failing

Got an intermittent issue failing builds for a project with git lfs enabled. Gitlab’s lfs server seems to be rejecting connections for some reason.

batch response: Unable to parse HTTP response for POST [https://gitlab-ci-token:[MASKED]@gitlab.orgname.com:9999/XXX/xxxxxx.git/info/lfs/objects/batch](https://gitlab-ci-token:[MASKED]@gitlab.orgname.com:9999/XXX/xxxxxx.git/info/lfs/objects/batch): http2: server sent GOAWAY and closed the connection; LastStreamID=1999, ErrCode=NO_ERROR, debug=""

[25](https://gitlab.orgname.com:9999/XXX/xxxxxx/-/jobs/250965#L25)error: failed to fetch some objects from '[https://gitlab-ci-token:[MASKED]@gitlab.orgname.com:9999/XXX/xxxxxx.git/info/lfs'](https://gitlab-ci-token:[MASKED]@gitlab.orgname.com:9999/XXX/xxxxxx.git/info/lfs')

The problem occurs frequently but not consistently. Usually re-running a job will see it go through. I’ve been looking for resource contention or some kind of rate limiting setting that might be happening somewhere but from what I can tell it’s mostly all default settings so I haven’t been able to find anything configuration wise.

We’re on self managed gitlab 13.11 hosted with docker, and using Kubernetes runners on version 14.1.
Could the version mismatch here be contributing? Anyone else come across anything like this before?

FWIW I never found the source of this problem. I managed to work around it by setting the GET_SOURCES_ATTEMPTS variable.

Following that we upgraded to 14.7 and I haven’t seen the issue trigger since. so ¯_(ツ)_/¯