I have gitlab-ce deployed to kubernetes using the helm chart, relevant config for LFS is below. When I try to push to LFS it acts like it is pushing, and then around 40MB resets to 0 in a loop before failing, with webservice returning a 500:
Logs from webservice:
Started PUT "/homelab/kubes.git/gitlab-lfs/objects/c4342215a34429252d8d85a44324e0dc7e96836d0c590c45e548a9c6740d9771/41512590" for 10.89.1.0 at 2022-10-12 21:13:22 +0000
Processing by Repositories::LfsStorageController#upload_finalize as HTML Parameters: {"repository_path"=>"homelab/kubes.git", "oid"=>"c4342215a34429252d8d85a44324e0dc7e96836d0c590c45e548a9c6740d9771", "size"=>"41512590"} Completed 500 Internal Server Error in 373ms (ActiveRecord: 11.0ms | Elasticsearch: 0.0ms | Allocations: 289004) JWT::DecodeError (Nil JSON web token): lib/gitlab/jwt_authenticatable.rb:21:in `decode_jwt' lib/gitlab/workhorse.rb:209:in `decode_jwt_with_issuer' lib/gitlab/workhorse.rb:205:in `verify_api_request!' app/controllers/concerns/workhorse_request.rb:13:in `verify_workhorse_api!' app/controllers/repositories/git_http_client_controller.rb:149:in `bypass_admin_mode!' app/controllers/application_controller.rb:531:in `set_current_admin' [...]
Fatal error: Server error: https://gitlab.xxx.com/homelab/kubes.git/gitlab-lfs/objects/c4342215a34429252d8d85a44324e0dc7e96836d0c590c45e548a9c6740d9771/41512590
Uploading LFS objects: 0% (0/1), 0 B | 2.4 MB/s, done.
error: failed to push some refs to 'git.xxx.com:homelab/kubes.git'
Config under global.appConfig:
object_store:
enabled: true
proxy_download: true
connection:
secret: gitlab-minio-auth
key: config
lfs:
enabled: true
bucket: gitlab-lfs
Buckets are in a minio deployment in the cluster (not deployed as a gitlab sub-chart, I already had minio), and this does work as things such as avatars are successfully stored and retrieved from the gitlab-uploads bucket and with the same config (consolidated object storage).
The only thing that might be worth noting is that web is https://gitlab.xxx.com and ssh is git.xxx.com – separate ingresses due to external load balancing for web connectivity. I can push/pull from either https using a token and ssh using an ssh key fine, so they do both work. Ingress is through an istio gateway which does tls termination but none of the gitlab pods are injected.
Any tips are greatly appreciated!