Hello everyone,
I’m unsure whether I’m hitting a bug or just some configuration errors. I run a self-hosted GitLab instance (v16.9.2-ee) via docker with s3 storage (minio) configured via the consolidated object storage config. The GitLab-s3 combination works well for all kinds of data storage (e.g. uploads, diffs, CI cache, pages, …) except for the dependency proxy.
When an image isn’t cached yet, it can be pulled without problems. On the second pull however, it fails with the following message in the docker client: Error response from daemon: missing or empty Content-Length header.
This is not too surprising because after the pull, the s3 bucket contains only a few blobs of some kilobytes. So all other blobs don’t get cached. When checking the logs from GitLab workhorse it’s obvious there are some errors with uploading the blobs:
{"code":500,"correlation_id":"01HRX788TTXXGPX600MGZY0ZZR","error":"dependency proxy: failed to upload file","level":"error","method":"GET","msg":"","time":"2024-03-14T00:54:34Z","uri":"/v2/software/dependency_proxy/containers/library/node/blobs/sha256:3cb8f9c23302e175d87a827f0a1c376bd59b1f6949bd3bc24ab8da0d669cdfa0"}
{"content_type":"application/octet-stream","correlation_id":"01HRX788TTXXGPX600MGZY0ZZR","duration_ms":860,"host":"git.domain","level":"info","method":"GET","msg":"access","proto":"HTTP/1.1","referrer":"","remote_addr":"[xxxx]:0","remote_ip":"xxxx","route":"","status":200,"system":"http","time":"2024-03-14T00:54:34Z","ttfb_ms":750,"uri":"/v2/software/dependency_proxy/containers/library/node/blobs/sha256:3cb8f9c23302e175d87a827f0a1c376bd59b1f6949bd3bc24ab8da0d669cdfa0","user_agent":"docker/25.0.4 go/go1.21.8 git-commit/061aa95 kernel/6.5.0-25-generic os/linux arch/amd64 UpstreamClient(Docker-Client/25.0.4 \\(linux\\))","written_bytes":1208623}
{"error":"MultipartUpload: upload multipart failed\n\tupload id: MzA1ODQxOGEtOTY5NS00MTMyLTgxYjctNTdlMGEzNDdlNjlmLjI1YWE1Y2UxLTUzZTQtNDAzYy04ZmFiLTk5NDZkMjlmOWQ2NQ\ncaused by: ReadRequestBody: read multipart upload data failed\ncaused by: write unix /var/opt/gitlab/gitlab-workhorse/sockets/socket-\u003e@: write: broken pipe","level":"error","msg":"error uploading S3 session","time":"2024-03-14T00:54:34Z"}
{"correlation_id":"01HRX788TTXXGPX600MG9ASBCE","error":"RequestBody: upload failed: write unix /var/opt/gitlab/gitlab-workhorse/sockets/socket-\u003e@: write: broken pipe","level":"error","method":"POST","msg":"","time":"2024-03-14T00:54:34Z","uri":""}
Any thoughts, pointers in the right direction and ideas to fix this are appreciated! 