Git lfs throwing Internal Server error in GitLab

Hi,

I have problem with git lfs and GitLab. Unfortunately, I am not sure whether the problem is in the git-lfs client (not sending the auth token) or in GitLab (not redirecting without the token). The error message in GitLab is:

Started POST "/core/product.git/info/lfs/objects/batch" for 10.0.3.1 at 2018-07-10 10:16:06 +0000
Processing by Projects::LfsApiController#batch as JSON
  Parameters: {"operation"=>"download", "objects"=>[{"oid"=>"be7a260a4cf2b416cf1cfc9cc06004b3d438573cae2a5258f0cfbe27f035d583", "size"=>19300120}, {"oid"=>"8d0784807b618bc1ab8d0ca53469c17020355336f26576dcc7e93644e8ccaad9", "size"=>14520360}, {"oid"=>"e575fe16c539a87fe48338b1c32439b6ab4a71086712ac966ad7ac32d6577ef9", "size"=>9659059}, {"oid"=>"29e76982d7e7caaaef7ddb28b9139c52d1d3f26bd3e3965754c255057c87d88b", "size"=>8605376}, {"oid"=>"03acf017b88ab6ad41d07f07fd7d5bd526b6b484e7a19e0ac3519c4fbdd74544", "size"=>8605368}, {"oid"=>"2bac9cc0991ea4b853e777b909dfef0f90450bf0dd2b946b59614510b93daeaa", "size"=>8230354}, {"oid"=>"2b3617fe6326c0648cc705c0cd31d1a1e7e8b6a2aee0f680547aa2273d3b570b", "size"=>8213331}, {"oid"=>"47069d9f8728ad3729a75a7d62b3f1b6f79e2dc38c06f9cc54a77a791da27458", "size"=>6478464}, {"oid"=>"71ea74c8ed5474b64bb9d9bb970157d8890c5249a77101c8e0d3dc3dd02caee9", "size"=>6457875}, {"oid"=>"3254e2fc37ac2d03bf35e7778c5bce082f0c38747d2401ea0d3bf86c88454a1b", "size"=>4171388}, {"oid"=>"7815d36e89aa6be488e8e99d60818dc12cdffc05a21633ecb719b222d897fe60", "size"=>4171388}, {"oid"=>"284b22ee0764db4692c02d0c5707573e0859f59b47d18244c6a3d0295fc3b5cd", "size"=>110922}, {"oid"=>"ac9e14773ffc97764566f0e2c66ca01104f1a97b187ded35d539cacaf5616d46", "size"=>110922}, {"oid"=>"53d5238c4b01be60a8b52fe59b282a5da98b6b3240afab6625f8bc7a0ee6158c", "size"=>110922}, {"oid"=>"bcea52ceef0c6a93a93bf495c07ade75d2bead0c2c6f2cde8c7cae596b9b13ef", "size"=>110922}, {"oid"=>"a969cefda0ec0b285f6ce5aeceb647e275162bcda6b652e5b107246bf0df627a", "size"=>24740}, {"oid"=>"95d3e6c1d88b3ea6d2031b3cd5166654698f921efd041eabd9ae5c58fccedb16", "size"=>2068}, {"oid"=>"ee1c07399e806213727dc8b0c492f2a259cf1dd25202f2a762e60a180a43a016", "size"=>209}, {"oid"=>"a02a27b1d1982c8516d83398e85a3c8b1aef1713c13ef4d84d7bde17430c07c4", "size"=>145}], "ref"=>{"name"=>"refs/heads/master"}, "namespace_id"=>"core", "project_id"=>"product.git", "lfs_api"=>{"operation"=>"download", "objects"=>[{"oid"=>"be7a260a4cf2b416cf1cfc9cc06004b3d438573cae2a5258f0cfbe27f035d583", "size"=>19300120}, {"oid"=>"8d0784807b618bc1ab8d0ca53469c17020355336f26576dcc7e93644e8ccaad9", "size"=>14520360}, {"oid"=>"e575fe16c539a87fe48338b1c32439b6ab4a71086712ac966ad7ac32d6577ef9", "size"=>9659059}, {"oid"=>"29e76982d7e7caaaef7ddb28b9139c52d1d3f26bd3e3965754c255057c87d88b", "size"=>8605376}, {"oid"=>"03acf017b88ab6ad41d07f07fd7d5bd526b6b484e7a19e0ac3519c4fbdd74544", "size"=>8605368}, {"oid"=>"2bac9cc0991ea4b853e777b909dfef0f90450bf0dd2b946b59614510b93daeaa", "size"=>8230354}, {"oid"=>"2b3617fe6326c0648cc705c0cd31d1a1e7e8b6a2aee0f680547aa2273d3b570b", "size"=>8213331}, {"oid"=>"47069d9f8728ad3729a75a7d62b3f1b6f79e2dc38c06f9cc54a77a791da27458", "size"=>6478464}, {"oid"=>"71ea74c8ed5474b64bb9d9bb970157d8890c5249a77101c8e0d3dc3dd02caee9", "size"=>6457875}, {"oid"=>"3254e2fc37ac2d03bf35e7778c5bce082f0c38747d2401ea0d3bf86c88454a1b", "size"=>4171388}, {"oid"=>"7815d36e89aa6be488e8e99d60818dc12cdffc05a21633ecb719b222d897fe60", "size"=>4171388}, {"oid"=>"284b22ee0764db4692c02d0c5707573e0859f59b47d18244c6a3d0295fc3b5cd", "size"=>110922}, {"oid"=>"ac9e14773ffc97764566f0e2c66ca01104f1a97b187ded35d539cacaf5616d46", "size"=>110922}, {"oid"=>"53d5238c4b01be60a8b52fe59b282a5da98b6b3240afab6625f8bc7a0ee6158c", "size"=>110922}, {"oid"=>"bcea52ceef0c6a93a93bf495c07ade75d2bead0c2c6f2cde8c7cae596b9b13ef", "size"=>110922}, {"oid"=>"a969cefda0ec0b285f6ce5aeceb647e275162bcda6b652e5b107246bf0df627a", "size"=>24740}, {"oid"=>"95d3e6c1d88b3ea6d2031b3cd5166654698f921efd041eabd9ae5c58fccedb16", "size"=>2068}, {"oid"=>"ee1c07399e806213727dc8b0c492f2a259cf1dd25202f2a762e60a180a43a016", "size"=>209}, {"oid"=>"a02a27b1d1982c8516d83398e85a3c8b1aef1713c13ef4d84d7bde17430c07c4", "size"=>145}], "ref"=>{"name"=>"refs/heads/master"}}}
Completed 500 Internal Server Error in 32ms (ActiveRecord: 6.3ms | Elasticsearch: 0.0ms)

NoMethodError (undefined method `auditor?' for #<DeployToken:0x00007fe5cfac1748>):
  app/policies/base_policy.rb:26:in `block in <class:BasePolicy>'
  lib/declarative_policy/condition.rb:21:in `instance_eval'
  lib/declarative_policy/condition.rb:21:in `compute'
  lib/declarative_policy/condition.rb:42:in `block in pass?'
  lib/declarative_policy/base.rb:280:in `cache'
  lib/declarative_policy/condition.rb:42:in `pass?'
  lib/declarative_policy/rule.rb:79:in `pass?'
  lib/declarative_policy/step.rb:79:in `pass?'
  lib/declarative_policy/runner.rb:89:in `block in run'
  lib/declarative_policy/runner.rb:177:in `block in steps_by_score'
  lib/declarative_policy/runner.rb:146:in `loop'
  lib/declarative_policy/runner.rb:146:in `steps_by_score'
  lib/declarative_policy/runner.rb:79:in `run'
  lib/declarative_policy/runner.rb:57:in `pass?'
  lib/declarative_policy/base.rb:215:in `block in allowed?'
  lib/declarative_policy/base.rb:215:in `each'
  lib/declarative_policy/base.rb:215:in `all?'
  lib/declarative_policy/base.rb:215:in `allowed?'
  lib/declarative_policy/rule.rb:157:in `pass?'
  lib/declarative_policy/step.rb:79:in `pass?'
  lib/declarative_policy/runner.rb:89:in `block in run'
  lib/declarative_policy/runner.rb:177:in `block in steps_by_score'
  lib/declarative_policy/runner.rb:146:in `loop'
  lib/declarative_policy/runner.rb:146:in `steps_by_score'
  lib/declarative_policy/runner.rb:79:in `run'
  lib/declarative_policy/runner.rb:57:in `pass?'
  lib/declarative_policy/base.rb:215:in `block in allowed?'
  lib/declarative_policy/base.rb:215:in `each'
  lib/declarative_policy/base.rb:215:in `all?'
  lib/declarative_policy/base.rb:215:in `allowed?'
  lib/declarative_policy/base.rb:207:in `can?'
  app/models/ability.rb:70:in `allowed?'
  app/controllers/application_controller.rb:141:in `can?'
  app/controllers/concerns/lfs_request.rb:91:in `user_can_download_code?'
  app/controllers/concerns/lfs_request.rb:75:in `lfs_download_access?'
  app/controllers/concerns/lfs_request.rb:36:in `lfs_check_access!'
  lib/gitlab/i18n.rb:50:in `with_locale'
  lib/gitlab/i18n.rb:56:in `with_user_locale'
  app/controllers/application_controller.rb:370:in `set_locale'
  lib/gitlab/middleware/multipart.rb:95:in `call'
  lib/gitlab/request_profiler/middleware.rb:14:in `call'
  ee/lib/gitlab/jira/middleware.rb:15:in `call'
  lib/gitlab/middleware/go.rb:17:in `call'
  lib/gitlab/etag_caching/middleware.rb:11:in `call'
  lib/gitlab/middleware/read_only/controller.rb:28:in `call'
  lib/gitlab/middleware/read_only.rb:16:in `call'
  lib/gitlab/request_context.rb:18:in `call'
  lib/gitlab/metrics/requests_rack_middleware.rb:27:in `call'
  lib/gitlab/middleware/release_env.rb:10:in `call'

If I use the ssh URL, it works fine:

gree@sana ~/desktop> git -c http.sslVerify=false clone git@<ourhost>.com:core/product.git
Cloning into 'product'...
remote: Counting objects: 3855, done.
remote: Compressing objects: 100% (266/266), done.
remote: Total 3855 (delta 280), reused 373 (delta 205)
Receiving objects: 100% (3855/3855), 54.48 MiB | 80.16 MiB/s, done.
Resolving deltas: 100% (1955/1955), done.
> POST /core/product.git/info/lfs/objects/batch HTTP/1.1
> Host: <ourhost>.com
> Accept: application/vnd.git-lfs+json; charset=utf-8
> Authorization: Basic * * * * *
> Content-Length: 1766
> Content-Type: application/vnd.git-lfs+json; charset=utf-8
> User-Agent: git-lfs/2.4.2 (GitHub; linux amd64; go 1.10.2; git 6f4b2e98)
>
{"operation":"download","objects":[{"oid":"be7a260a4cf2b416cf1cfc9cc06004b3d438573cae2a5258f0cfbe27f035d583","size":19300120},{"oid":"8d0784807b618bc1ab8d0ca53469c17020355336f26576dcc7e93644e8ccaad9","size":14520360},{"oid":"e575fe16c539a87fe48338b1c32439b6ab4a71086712ac966ad7ac32d6577ef9","size":9659059},{"oid":"29e76982d7e7caaaef7ddb28b9139c52d1d3f26bd3e3965754c255057c87d88b","size":8605376},{"oid":"03acf017b88ab6ad41d07f07fd7d5bd526b6b484e7a19e0ac3519c4fbdd74544","size":8605368},{"oid":"2bac9cc0991ea4b853e777b909dfef0f90450bf0dd2b946b59614510b93daeaa","size":8230354},{"oid":"2b3617fe6326c0648cc705c0cd31d1a1e7e8b6a2aee0f680547aa2273d3b570b","size":8213331},{"oid":"47069d9f8728ad3729a75a7d62b3f1b6f79e2dc38c06f9cc54a77a791da27458","size":6478464},{"oid":"71ea74c8ed5474b64bb9d9bb970157d8890c5249a77101c8e0d3dc3dd02caee9","size":6457875},{"oid":"3254e2fc37ac2d03bf35e7778c5bce082f0c38747d2401ea0d3bf86c88454a1b","size":4171388},{"oid":"7815d36e89aa6be488e8e99d60818dc12cdffc05a21633ecb719b222d897fe60","size":4171388},{"oid":"284b22ee0764db4692c02d0c5707573e0859f59b47d18244c6a3d0295fc3b5cd","size":110922},{"oid":"ac9e14773ffc97764566f0e2c66ca01104f1a97b187ded35d539cacaf5616d46","size":110922},{"oid":"53d5238c4b01be60a8b52fe59b282a5da98b6b3240afab6625f8bc7a0ee6158c","size":110922},{"oid":"bcea52ceef0c6a93a93bf495c07ade75d2bead0c2c6f2cde8c7cae596b9b13ef","size":110922},{"oid":"a969cefda0ec0b285f6ce5aeceb647e275162bcda6b652e5b107246bf0df627a","size":24740},{"oid":"95d3e6c1d88b3ea6d2031b3cd5166654698f921efd041eabd9ae5c58fccedb16","size":2068},{"oid":"ee1c07399e806213727dc8b0c492f2a259cf1dd25202f2a762e60a180a43a016","size":209},{"oid":"a02a27b1d1982c8516d83398e85a3c8b1aef1713c13ef4d84d7bde17430c07c4","size":145}],"ref":{"name":"refs/heads/master"}}

< HTTP/1.1 200 OK
< Transfer-Encoding: chunked
< Access-Control-Allow-Credentials: true
< Access-Control-Allow-Headers: accept, content-type
< Access-Control-Allow-Methods: GET, POST, OPTIONS
< Access-Control-Allow-Origin: *
< Cache-Control: max-age=0, private, must-revalidate
< Content-Type: application/json; charset=utf-8
< Date: Tue, 10 Jul 2018 09:17:47 GMT
< Etag: W/"01c486ae97bc20f47256e4239bdf28b6"
< Page-Title: GitLab
< Server: nginx/1.10.3
< Strict-Transport-Security: max-age=31536000
< Vary: Accept-Encoding
< X-Content-Type-Options: nosniff
< X-Frame-Options: DENY
< X-Request-Id: b326db46-2220-4d82-ad4c-fd8f09cec6a0
< X-Runtime: 0.051248
< X-Ua-Compatible: IE=edge
< X-Xss-Protection: 1; mode=block

But using https, it throws the internal server error:

gree@sana ~/desktop> git -c http.sslVerify=false clone https://gitlab+deploy-token-1:<anonymized>@<ourhost>.com/core/product.git
Cloning into 'product'...
* Couldn't find host <ourhost>.com in the .netrc file; using defaults
*   Trying 178.250.146.250...
* Connected to <ourhost>.com (178.250.146.250) port 443 (#0)
* found 148 certificates in /etc/ssl/certs/ca-certificates.crt
* found 597 certificates in /etc/ssl/certs
* ALPN, offering http/1.1
* SSL connection using TLS1.2 / ECDHE_RSA_AES_128_GCM_SHA256
*        server certificate verification SKIPPED
*        server certificate status verification SKIPPED
*        common name: *.stcorp.nl (does not match '<ourhost>.com')
*        server certificate expiration date OK
*        server certificate activation date OK
*        certificate public key: RSA
*        certificate version: #3
*        subject: CN=*.stcorp.nl
*        start date: Wed, 25 Apr 2018 00:00:00 GMT
*        expire date: Mon, 27 Apr 2020 12:00:00 GMT
*        issuer: C=US,O=DigiCert Inc,OU=www.digicert.com,CN=RapidSSL TLS RSA CA G1
*        compression: NULL
* ALPN, server accepted to use http/1.1
> GET /core/product.git/info/refs?service=git-upload-pack HTTP/1.1
Host: <ourhost>.com
User-Agent: git/2.18.0
Accept: */*
Accept-Encoding: deflate, gzip
Accept-Language: en-US, cs;q=0.9, *;q=0.8
Pragma: no-cache

< HTTP/1.1 401 Unauthorized
< Date: Tue, 10 Jul 2018 10:16:04 GMT
< Server: nginx/1.10.3
< Content-Type: text/plain; charset=utf-8
< Content-Length: 26
< Cache-Control: no-cache
< WWW-Authenticate: Basic realm="GitLab"
< X-Content-Type-Options: nosniff
< X-Frame-Options: DENY
< X-Request-Id: 926f8983-2927-4526-a869-728a449c2506
< X-Runtime: 0.015799
< X-Ua-Compatible: IE=edge
< X-Xss-Protection: 1; mode=block
<
* Ignoring the response-body
* Connection #0 to host <ourhost>.com left intact
* Issue another request to this URL: 'https://gitlab+deploy-token-1:<anonymized>@<ourhost>.com/core/product.git/info/refs?service=git-upload-pack'
* Couldn't find host <ourhost>.com in the .netrc file; using defaults
* Found bundle for host <ourhost>.com: 0x10873f0 [can pipeline]
* Re-using existing connection! (#0) with host <ourhost>.com
* Connected to <ourhost>.com (178.250.146.250) port 443 (#0)
* Server auth using Basic with user 'gitlab+deploy-token-1'
> GET /core/product.git/info/refs?service=git-upload-pack HTTP/1.1
Host: <ourhost>.com
Authorization: Basic Z2l0bGFiK2RlcGxveS10b2tlbi0xOnViWlB5YzJIRVRxRUtvdTlwcERj
User-Agent: git/2.18.0
Accept: */*
Accept-Encoding: deflate, gzip
Accept-Language: en-US, cs;q=0.9, *;q=0.8
Pragma: no-cache

< HTTP/1.1 200 OK
< Date: Tue, 10 Jul 2018 10:16:04 GMT
< Server: nginx/1.10.3
< Content-Type: application/x-git-upload-pack-advertisement
< Content-Length: 572
< Cache-Control: no-cache
< Strict-Transport-Security: max-age=31536000
< Access-Control-Allow-Methods: GET, POST, OPTIONS
< Access-Control-Allow-Origin: *
< Access-Control-Allow-Credentials: true
< Access-Control-Allow-Headers: accept, content-type
<
* Connection #0 to host <ourhost>.com left intact
* Couldn't find host <ourhost>.com in the .netrc file; using defaults
* Found bundle for host <ourhost>.com: 0x10873f0 [can pipeline]
* Re-using existing connection! (#0) with host <ourhost>.com
* Connected to <ourhost>.com (178.250.146.250) port 443 (#0)
* Server auth using Basic with user 'gitlab+deploy-token-1'
> POST /core/product.git/git-upload-pack HTTP/1.1
Host: <ourhost>.com
Authorization: Basic Z2l0bGFiK2RlcGxveS10b2tlbi0xOnViWlB5YzJIRVRxRUtvdTlwcERj
User-Agent: git/2.18.0
Accept-Encoding: deflate, gzip
Content-Type: application/x-git-upload-pack-request
Accept: application/x-git-upload-pack-result
Content-Length: 315

* upload completely sent off: 315 out of 315 bytes
< HTTP/1.1 200 OK
< Date: Tue, 10 Jul 2018 10:16:04 GMT
< Server: nginx/1.10.3
< Content-Type: application/x-git-upload-pack-result
< Cache-Control: no-cache
< Strict-Transport-Security: max-age=31536000
< Access-Control-Allow-Methods: GET, POST, OPTIONS
< Access-Control-Allow-Origin: *
< Access-Control-Allow-Credentials: true
< Access-Control-Allow-Headers: accept, content-type
< Transfer-Encoding: chunked
<
remote: Counting objects: 3855, done.
remote: Compressing objects: 100% (266/266), done.
remote: Total 3855 (delta 280), reused 373 (delta 205)
* Connection #0 to host <ourhost>.com left intact
Receiving objects: 100% (3855/3855), 54.48 MiB | 37.47 MiB/s, done.
Resolving deltas: 100% (1955/1955), done.
> POST /core/product.git/info/lfs/objects/batch HTTP/1.1
> Host: <ourhost>.com
> Accept: application/vnd.git-lfs+json; charset=utf-8
> Content-Length: 1766
> Content-Type: application/vnd.git-lfs+json; charset=utf-8
> User-Agent: git-lfs/2.4.2 (GitHub; linux amd64; go 1.10.2; git 6f4b2e98)
>
{"operation":"download","objects":[{"oid":"be7a260a4cf2b416cf1cfc9cc06004b3d438573cae2a5258f0cfbe27f035d583","size":19300120},{"oid":"8d0784807b618bc1ab8d0ca53469c17020355336f26576dcc7e93644e8ccaad9","size":14520360},{"oid":"e575fe16c539a87fe48338b1c32439b6ab4a71086712ac966ad7ac32d6577ef9","size":9659059},{"oid":"29e76982d7e7caaaef7ddb28b9139c52d1d3f26bd3e3965754c255057c87d88b","size":8605376},{"oid":"03acf017b88ab6ad41d07f07fd7d5bd526b6b484e7a19e0ac3519c4fbdd74544","size":8605368},{"oid":"2bac9cc0991ea4b853e777b909dfef0f90450bf0dd2b946b59614510b93daeaa","size":8230354},{"oid":"2b3617fe6326c0648cc705c0cd31d1a1e7e8b6a2aee0f680547aa2273d3b570b","size":8213331},{"oid":"47069d9f8728ad3729a75a7d62b3f1b6f79e2dc38c06f9cc54a77a791da27458","size":6478464},{"oid":"71ea74c8ed5474b64bb9d9bb970157d8890c5249a77101c8e0d3dc3dd02caee9","size":6457875},{"oid":"3254e2fc37ac2d03bf35e7778c5bce082f0c38747d2401ea0d3bf86c88454a1b","size":4171388},{"oid":"7815d36e89aa6be488e8e99d60818dc12cdffc05a21633ecb719b222d897fe60","size":4171388},{"oid":"284b22ee0764db4692c02d0c5707573e0859f59b47d18244c6a3d0295fc3b5cd","size":110922},{"oid":"ac9e14773ffc97764566f0e2c66ca01104f1a97b187ded35d539cacaf5616d46","size":110922},{"oid":"53d5238c4b01be60a8b52fe59b282a5da98b6b3240afab6625f8bc7a0ee6158c","size":110922},{"oid":"bcea52ceef0c6a93a93bf495c07ade75d2bead0c2c6f2cde8c7cae596b9b13ef","size":110922},{"oid":"a969cefda0ec0b285f6ce5aeceb647e275162bcda6b652e5b107246bf0df627a","size":24740},{"oid":"95d3e6c1d88b3ea6d2031b3cd5166654698f921efd041eabd9ae5c58fccedb16","size":2068},{"oid":"ee1c07399e806213727dc8b0c492f2a259cf1dd25202f2a762e60a180a43a016","size":209},{"oid":"a02a27b1d1982c8516d83398e85a3c8b1aef1713c13ef4d84d7bde17430c07c4","size":145}],"ref":{"name":"refs/heads/master"}}

< HTTP/1.1 500 Internal Server Error
< Connection: close
< Content-Length: 2902
< Cache-Control: no-cache, no-store, max-age=0, must-revalidate
< Content-Type: text/html; charset=utf-8
< Date: Tue, 10 Jul 2018 10:16:06 GMT
< Expires: Mon, 01 Jan 1990 00:00:00 GMT
< Pragma: no-cache
< Server: nginx/1.10.3
< X-Request-Id: 302d6940-6564-4f30-9feb-ea29578ba4bf
< X-Runtime: 0.063416

Thanks for any help,

Antonin

Hi, I am have the same problem.
In my case I try to clone git repo with LFS-files using deploy token.
I use GitLab 11.1.4-ee.

For temporary workaround here is patch that fix error:

--- a/app/policies/base_policy.rb 2018-08-06 20:16:53.426533831 +0700
+++ b/app/policies/base_policy.rb 2018-08-06 20:15:20.128351511 +0700
@@ -5,7 +5,7 @@
 
   desc "User is an instance admin"
   with_options scope: :user, score: 0
-  condition(:admin) { @user&.admin? }
+  condition(:admin) { @user && @user.class.method_defined?(:admin?) && @user&.admin? }
 
   with_options scope: :user, score: 0
   condition(:external_user) { @user.nil? || @user.external? }
@@ -23,7 +23,7 @@
 
   # EE Extensions
   with_scope :user
-  condition(:auditor, score: 0) { @user&.auditor? }
+  condition(:auditor, score: 0) { @user && @user.class.method_defined?(:auditor?) && @user&.auditor? }
 
   with_scope :user
   condition(:support_bot, score: 0) { @user&.support_bot? }

Hi Anton, the patch works. Thanks a lot. Should I file a bug report? Or will you?