Hi,
I have problem with git lfs, I push and return 500 error.
gitlab version: 12.2.5
platform: docker-compose
git lfs version: 2.8.0
GIT_TRACE=1 GIT_CURL_VERBOSE=1 git push origin master
15:33:22.809462 git.c:419 trace: built-in: git push origin master
15:33:22.814500 run-command.c:643 trace: run_command: GIT_DIR=.git git-remote-https origin https://test-lfs.xxxx.com:10443/root/first-lfs.git
* Couldn’t find host test-lfs.xxxx.com in the .netrc file; using defaults
* Trying xxxx…
* TCP_NODELAY set
* Connected to test-lfs.xxxx.com (xxxx) port 10443 (#0)
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
* CAfile: /etc/ssl/cert.pem
CApath: none
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server accepted to use http/1.1
* Server certificate:
* start date: Nov 25 00:00:00 2019 GMT
* expire date: Jan 23 12:00:00 2021 GMT
* subjectAltName: host “test-lfs.xxxx.com” matched cert’s “*.xxxx.com”
* issuer: C=US; O=DigiCert Inc; OU=www.digicert.com; CN=GeoTrust CN RSA CA G1
* SSL certificate verify ok.
> GET /root/first-lfs.git/info/refs?service=git-receive-pack HTTP/1.1
Host: test-lfs.xxxx.com:10443
User-Agent: git/2.21.0
Accept: /
Accept-Encoding: deflate, gzip
Accept-Language: zh-CN, *;q=0.9
Pragma: no-cache
< HTTP/1.1 401 Unauthorized
< Server: nginx
< Date: Mon, 30 Dec 2019 07:33:23 GMT
< Content-Type: text/plain; charset=utf-8
< Content-Length: 26
< Connection: keep-alive
< Cache-Control: no-cache
< Referrer-Policy: strict-origin-when-cross-origin
< Set-Cookie: experimentation_subject_id=IjgwOWM0ZjFhLTEyNzQtNDJlZi04ZTMwLWMzYzZmNTQyYzQxMCI%3D--a089a7dd9963ba5afc68eb7861e4cfdbfe272f08; domain=.xxxx.com; path=/; expires=Fri, 30 Dec 2039 07:33:23 -0000; secure; HttpOnly
< WWW-Authenticate: Basic realm="GitLab"
< X-Content-Type-Options: nosniff
< X-Download-Options: noopen
< X-Frame-Options: DENY
< X-Permitted-Cross-Domain-Policies: none
< X-Request-Id: 7f888b79-c56d-484d-bc2c-e310922b8e0b
< X-Runtime: 0.040879
< X-Ua-Compatible: IE=edge
< X-Xss-Protection: 1; mode=block
<
* Connection #0 to host test-lfs.xxxx.com left intact
15:33:23.675227 run-command.c:643 trace: run_command: 'git credential-osxkeychain get'
15:33:23.686509 git.c:670 trace: exec: git-credential-osxkeychain get
15:33:23.687109 run-command.c:643 trace: run_command: git-credential-osxkeychain get
* Couldn't find host test-lfs.xxxx.com in the .netrc file; using defaults
* Found bundle for host test-lfs.xxxx.com: 0x7f9ba541bb50 [can pipeline]
* Re-using existing connection! (#0) with host test-lfs.xxxx.com
* Connected to test-lfs.xxxx.com (xxxx) port 10443 (#0)
* Server auth using Basic with user 'root'
> GET /root/first-lfs.git/info/refs?service=git-receive-pack HTTP/1.1
Host: test-lfs.xxxx.com:10443
Authorization: Basic cm9vdDprbGFiMjAxNw==
User-Agent: git/2.21.0
Accept: */*
Accept-Encoding: deflate, gzip
Accept-Language: zh-CN, *;q=0.9
Pragma: no-cache
< HTTP/1.1 200 OK
< Server: nginx
< Date: Mon, 30 Dec 2019 07:33:23 GMT
< Content-Type: application/x-git-receive-pack-advertisement
< Content-Length: 196
< Connection: keep-alive
< Cache-Control: no-cache
< Content-Encoding: gzip
< X-Accel-Buffering: no
< Strict-Transport-Security: max-age=31536000;
<
* Connection #0 to host test-lfs.xxxx.com left intact
15:33:24.088822 run-command.c:643 trace: run_command: 'git credential-osxkeychain store'
15:33:24.100923 git.c:670 trace: exec: git-credential-osxkeychain store
15:33:24.101450 run-command.c:643 trace: run_command: git-credential-osxkeychain store
15:33:24.157823 run-command.c:643 trace: run_command: 'git credential-cache --timeout=3600 store'
15:33:24.166166 git.c:670 trace: exec: git-credential-cache --timeout=3600 store
15:33:24.166617 run-command.c:643 trace: run_command: git-credential-cache --timeout=3600 store
15:33:24.186593 run-command.c:643 trace: run_command: .git/hooks/pre-push origin https://test-lfs.xxxx.com:10443/root/first-lfs.git
15:33:24.215519 git.c:670 trace: exec: git-lfs pre-push origin https://test-lfs.xxxx.com:10443/root/first-lfs.git
15:33:24.221316 run-command.c:643 trace: run_command: git-lfs pre-push origin https://test-lfs.xxxx.com:10443/root/first-lfs.git
15:33:24.286269 trace git-lfs: exec: git 'version'
15:33:24.304594 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'rev-parse' 'HEAD' '--symbolic-full-name' 'HEAD'
15:33:24.316169 trace git-lfs: exec: git 'config' '-l'
15:33:24.323167 trace git-lfs: pre-push: refs/heads/master dcef9eecc55c8bd1bbd8ac0158731df947134623 refs/heads/master 5ba454e932321f68f1f19748784bf6aab2319cff
15:33:25.446801 trace git-lfs: creds: git credential fill ("https", "test-lfs.xxxx.com:10443", "")
15:33:25.454790 git.c:419 trace: built-in: git credential fill
15:33:25.455724 run-command.c:643 trace: run_command: 'git credential-osxkeychain get'
15:33:25.469442 git.c:670 trace: exec: git-credential-osxkeychain get
15:33:25.469951 run-command.c:643 trace: run_command: git-credential-osxkeychain get
15:33:25.517997 trace git-lfs: Filled credentials for https://test-lfs.xxxx.com:10443/root/first-lfs.git
15:33:25.604592 trace git-lfs: HTTP: POST https://test-lfs.xxxx.com:10443/root/first-lfs.git/info/lfs/locks/verify
> POST /root/first-lfs.git/info/lfs/locks/verify HTTP/1.1
> Host: test-lfs.xxxx.com:10443
> Accept: application/vnd.git-lfs+json; charset=utf-8
> Authorization: Basic cm9vdDprbGFiMjAxNw==
> Content-Length: 36
> Content-Type: application/vnd.git-lfs+json; charset=utf-8
> User-Agent: git-lfs/2.8.0 (GitHub; darwin amd64; go 1.12.7)
>
{"ref":{"name":"refs/heads/master"}}15:33:26.935404 trace git-lfs: HTTP: 200
< HTTP/1.1 200 OK
< Content-Length: 23
< Cache-Control: max-age=0, private, must-revalidate
< Connection: keep-alive
< Content-Type: application/vnd.git-lfs+json; charset=utf-8
< Date: Mon, 30 Dec 2019 07:33:26 GMT
< Etag: W/"928ac95441b0331342f5e9d1e1150f67"
< Page-Title: GitLab
< Referrer-Policy: strict-origin-when-cross-origin
< Server: nginx
< Set-Cookie: experimentation_subject_id=ImYwMGNiYzc4LWJlYjItNGI4NS05ZGQxLTc2ZGNmZTI3NTM3NSI%3D--0f5db4311a9bf4381da1beba87bd9a714940adfa; domain=.xxxx.com; path=/; expires=Fri, 30 Dec 2039 07:33:26 -0000; secure; HttpOnly
< Strict-Transport-Security: max-age=31536000;
< X-Accel-Buffering: no
< X-Content-Type-Options: nosniff
< X-Download-Options: noopen
< X-Frame-Options: DENY
< X-Permitted-Cross-Domain-Policies: none
< X-Request-Id: ndGLYvhrZ32
< X-Runtime: 0.097735
< X-Ua-Compatible: IE=edge
< X-Xss-Protection: 1; mode=block
<
15:33:26.935574 trace git-lfs: creds: git credential approve ("https", "test-lfs.xxxx.com:10443", "")
15:33:26.943652 git.c:419 trace: built-in: git credential approve
15:33:26.944586 run-command.c:643 trace: run_command: 'git credential-osxkeychain store'
15:33:26.953421 git.c:670 trace: exec: git-credential-osxkeychain store
15:33:26.953986 run-command.c:643 trace: run_command: git-credential-osxkeychain store
15:33:26.999111 run-command.c:643 trace: run_command: 'git credential-cache --timeout=3600 store'
15:33:27.010120 git.c:670 trace: exec: git-credential-cache --timeout=3600 store
15:33:27.010701 run-command.c:643 trace: run_command: git-credential-cache --timeout=3600 store
15:33:27.025680 trace git-lfs: HTTP: {"ours":[],"theirs":[]}
{"ours":[],"theirs":[]}Locking support detected on remote "origin". Consider enabling it with:
$ git config lfs.https://test-lfs.xxxx.com:10443/root/first-lfs.git/info/lfs.locksverify true
15:33:27.026250 trace git-lfs: tq: running as batched queue, batch size of 100
15:33:27.026492 trace git-lfs: run_command: git rev-list --stdin --objects --not --remotes=origin --
15:33:27.041707 trace git-lfs: tq: sending batch of size 1
15:33:27.042444 trace git-lfs: api: batch 1 files
15:33:27.042578 trace git-lfs: creds: git credential cache ("https", "test-lfs.xxxx.com:10443", "")
15:33:27.042587 trace git-lfs: Filled credentials for https://test-lfs.xxxx.com:10443/root/first-lfs.git
15:33:27.042600 trace git-lfs: HTTP: POST https://test-lfs.xxxx.com:10443/root/first-lfs.git/info/lfs/objects/batch
> POST /root/first-lfs.git/info/lfs/objects/batch HTTP/1.1
> Host: test-lfs.xxxx.com:10443
> Accept: application/vnd.git-lfs+json; charset=utf-8
> Authorization: Basic cm9vdDprbGFiMjAxNw==
> Content-Length: 157
> Content-Type: application/vnd.git-lfs+json; charset=utf-8
> User-Agent: git-lfs/2.8.0 (GitHub; darwin amd64; go 1.12.7)
>
{"operation":"upload","objects":[{"oid":"210d848060fac835a0e4202c96c78afcf7665a4efa2057cf674d0ef0fcda85eb","size":75972}],"ref":{"name":"refs/heads/master"}}15:33:27.295979 trace git-lfs: HTTP: 200
< HTTP/1.1 200 OK
< Content-Length: 687
< Cache-Control: max-age=0, private, must-revalidate
< Connection: keep-alive
< Content-Type: application/json; charset=utf-8
< Date: Mon, 30 Dec 2019 07:33:27 GMT
< Etag: W/"08e21c4bb072d133aa81476bc2ae5939"
< Page-Title: GitLab
< Referrer-Policy: strict-origin-when-cross-origin
< Server: nginx
< Set-Cookie: experimentation_subject_id=ImJiOGZmMTcxLWU4OGYtNDYwOS1hNjQ2LWRkMDYwZGE5MjU1ZiI%3D--224d2f0c79bf63cba9f9036702d69f240f9d9e5b; domain=.xxxx.com; path=/; expires=Fri, 30 Dec 2039 07:33:27 -0000; secure; HttpOnly
< Strict-Transport-Security: max-age=31536000;
< X-Accel-Buffering: no
< X-Content-Type-Options: nosniff
< X-Download-Options: noopen
< X-Frame-Options: DENY
< X-Permitted-Cross-Domain-Policies: none
< X-Request-Id: LzcBoVaZBq7
< X-Runtime: 0.099640
< X-Ua-Compatible: IE=edge
< X-Xss-Protection: 1; mode=block
<
15:33:27.296701 trace git-lfs: HTTP: {"objects":[{"oid":"210d848060fac835a0e4202c96c78afcf7665a4efa2057cf674d0ef0fcda85eb","size":75972,"actions":{"upload":{"href":"https://test-lfs.xxxx.com:10443/root/first-lfs.git/gitlab-lfs/objects/210d848060fac835a0e4202c96c78afcf7665a4efa2057cf674d0ef0fcda85eb/75972","header":{"Authorization":"Basic cm9vdDpleUpoYkdjaU9pSklVekkxTmluUjVjQ0k2SWtwWFZDSjkuZXlKa1lYUmhJanA3SW1GamRHOXlJam9pY205dmRDSjlMQ0pxZEdraU9pSTJPRGxsTmpsbU5TMDFPVEkxTFRSa09UY3RZV0V6TlMwMllUa3hOekJtTTJGbU5qY2lMQ0pwWVhRaU9qRTFOemMyT1RFeU
{"objects":[{"oid":"210d848060fac835a0e4202c96c78afcf7665a4efa2057cf674d0ef0fcda85eb","size":75972,"actions":{"upload":{"href":"https://test-lfs.xxxx.com:10443/root/first-lfs.git/gitlab-lfs/objects/210d848060fac835a0e4202c96c78afcf7665a4efa2057cf674d0ef0fcda85eb/75972","header":{"Authorization":"Basic cm9vdDpleUpoYkdjaU9pSkkxTmlJc0luUjVjQ0k2SWtwWFZDSjkuZXlKa1lYUmhJanA3SW1GamRHOXlJam9pY205dmRDSjlMQ0pxZEdraU9pSTJPRGxsTmpsbU5TMDFPVEkxTFRSa09UY3RZV0V6TlMwMllUa3hOekJtTTJGbU5qY2lMQ0pwWVhRaU9qRTFOemMyT1RFeU15:33:27.297215 trace git-lfs: HTTP: 1EY3NJbTVpWmlJNk1UVTNOelk1TVRJd01pd2laWGh3SWpveE5UYzNOamt6TURBM2ZRLkFfdXpSR2JnaGw2RDZoT2Ywb3pzdW9fT1Y0b1Y0Y2lwTGI1TzRIWlU=","Content-Type":"application/octet-stream"}}}}]}
1EY3NJbTVpWmlJNk1UVTNOelk1TVRJd01pd2laWGh3SWpveE5UYzNOamt6TURBM2ZRLkFfdXpSR2JnaGw2RDZwSbYwb3pzdW9fT1Y0b1Y0Y2lwTGI1TzRIWlU=","Content-Type":"application/octet-stream"}}}}]}15:33:27.298335 trace git-lfs: tq: starting transfer adapter "basic"
15:33:27.298623 trace git-lfs: xfer: adapter "basic" Begin() with 8 workers
15:33:27.298690 trace git-lfs: xfer: adapter "basic" started
15:33:27.298736 trace git-lfs: xfer: adapter "basic" worker 3 starting
15:33:27.298899 trace git-lfs: xfer: adapter "basic" worker 3 waiting for Auth
15:33:27.298971 trace git-lfs: xfer: adapter "basic" worker 0 starting
15:33:27.298997 trace git-lfs: xfer: adapter "basic" worker 4 starting
15:33:27.299022 trace git-lfs: xfer: adapter "basic" worker 4 waiting for Auth
15:33:27.299003 trace git-lfs: xfer: adapter "basic" worker 0 processing job for "210d848060fac835a0e4202c96c78afcf7665a4efa2057cf674d0ef0fcda85eb"
15:33:27.298828 trace git-lfs: xfer: adapter "basic" worker 7 starting
15:33:27.299119 trace git-lfs: xfer: adapter "basic" worker 7 waiting for Auth
15:33:27.299004 trace git-lfs: xfer: adapter "basic" worker 1 starting
15:33:27.299313 trace git-lfs: xfer: adapter "basic" worker 1 waiting for Auth
15:33:27.299038 trace git-lfs: xfer: adapter "basic" worker 6 starting
15:33:27.299506 trace git-lfs: xfer: adapter "basic" worker 6 waiting for Auth
15:33:27.298986 trace git-lfs: xfer: adapter "basic" worker 5 starting
15:33:27.299616 trace git-lfs: xfer: adapter "basic" worker 5 waiting for Auth
15:33:27.299418 trace git-lfs: xfer: adapter "basic" worker 2 starting
15:33:27.299741 trace git-lfs: xfer: adapter "basic" worker 2 waiting for Auth
15:33:27.301737 trace git-lfs: HTTP: PUT https://test-lfs.xxxx.com:10443/root/first-lfs.git/gitlab-lfs/objects/210d848060fac835a0e4202c96c78afcf7665a4efa2057cf674d0ef0fcda85eb/75972
> PUT /root/first-lfs.git/gitlab-lfs/objects/210d848060fac835a0e4202c96c78afcf7665a4efa2057cf674d0ef0fcda85eb/75972 HTTP/1.1
> Host: test-lfs.xxxx.com:10443
> Authorization: Basic cm9vdDpleUpoYkdjaU9pSklVekkxTmlJc0luUjVjQ0k2SWtwWFZDSjkuZXlKamhJanA3SW1GamRHOXlJam9pY205dmRDSjlMQ0pxZEdraU9pSTJPRGxsTmpsbU5TMDFPVEkxTFRSa09UY3RZV0V6TlMwMllUa3hOekJtTTJGbU5qY2lMQ0pwWVhRaU9qRTFOemMyT1RFeU1EY3NJbTVpWmlJNk1UVTNOelk1TVRJd01pd2laWGh3SWpveE5UYzNOamt6TURBM2ZRLXpSR2JnaGw2RDZwSGJoT2Ywb3pzdW9fT1Y0b1Y0Y2lwTGI1TzRIWlU=
> Content-Length: 75972
> Content-Type: application/octet-stream
> User-Agent: git-lfs/2.8.0 (GitHub; darwin amd64; go 1.12.7)
>
15:33:27.302100 trace git-lfs: xfer: adapter "basic" worker 7 auth signal received
15:33:27.302162 trace git-lfs: xfer: adapter "basic" worker 1 auth signal received
15:33:27.302158 trace git-lfs: xfer: adapter "basic" worker 2 auth signal received
15:33:27.302167 trace git-lfs: xfer: adapter "basic" worker 6 auth signal received
15:33:27.302171 trace git-lfs: xfer: adapter "basic" worker 5 auth signal received
15:33:27.302172 trace git-lfs: xfer: adapter "basic" worker 3 auth signal received
15:33:27.302173 trace git-lfs: xfer: adapter "basic" worker 4 auth signal received
15:33:28.301920 trace git-lfs: HTTP: 500
< HTTP/1.1 500 Internal Server Error
< Content-Length: 22
< Connection: keep-alive
< Content-Type: text/plain; charset=utf-8
< Date: Mon, 30 Dec 2019 07:33:28 GMT
< Server: nginx
< X-Content-Type-Options: nosniff
and the error log from gitlab-workhorse
time=“2019-12-30T07:33:30Z” level=error msg=error correlation_id=ktRySMrkrj6 error=“BodyUploader: upload failed: PUT request “//s3.amazon.com/kubernetes-lfs/tmp/uploads/1577691210-2464-0018-8357-e306a399303db9af2d201f55faa209?X-Amz-Expires=15300&X-Amz-Date=20191230T03330Z&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAP74IPLHCVUBA%2F20191230%2Fs3-north-1%2Fs3%2Faws4_request&X-Amz-SignedHeaders=host&X-Amz-Signature=[FILTERED]”: Put //s3.amazon.com/kubernetes-lfs/tmp/uploads/1577691210-2464-0018-8357-e306a399303db9af2d20f55fa67a209?X-Amz-Expires=15300&X-Amz-Date=2019120T073330Z&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=A%2F20191230%2Fcn-northwest-1%2Fs3%2Faws4_request&X-Amz-SignedHeaders=host&X-Amz-Signature=41b7171e5a6f2811cc9b9a6745629139f37ac410a18586d0798ff8cc9: unsupported protocol scheme “”” method=PUT uri=/root/first-lfs.git/gitlab-lfs/objects/210d848060fac835a0e4202c96c78afcf7665a4efa2057cf674d0ef0fcda85eb/75972