WARNING: Uploading artifacts as "archive" to coordinator... POST.. 500 Internal Server Error

How to Use GitLab
1

Replace this template with your information

Describe your question in as much detail as possible:

We are running self-managed Gitlab Helm charts 15.9.2 on-prem. We have been contacted by a user that receives this error:

WARNING: Uploading artifacts as "archive" to coordinator... POST.. 500 Internal Server Error

Before that we incresed the artifacts size to 10G. I tailed the logs and found this error:

time="2023-03-24T12:26:15Z" level=error msg="Unable to write to client." cause="write tcp 10.244.1.126:9000->10.244.1.80:40678: write: broken pipe" source="[object-handlers.go:187:objectAPIHandlers.GetObjectHandler()]" stack="/q/.q/sources/gopath/src/github.com/minio/minio/cmd/fs-v1.go:450:fsObjects.GetObject() <autogenerated>:1:(*fsObjects).GetObject() /q/.q/sources/gopath/src/github.com/minio/minio/cmd/object-handlers.go:186:objectAPIHandlers.GetObjectHandler() /q/.q/sources/gopath/src/github.com/minio/minio/cmd/api-router.go:59:(objectAPIHandlers).GetObjectHandler-fm() /opt/go/src/net/http/server.go:1918:HandlerFunc.ServeHTTP() /q/.q/sources/gopath/src/github.com/minio/minio/vendor/github.com/gorilla/mux/mux.go:107:(*Router).ServeHTTP() /q/.q/sources/gopath/src/github.com/minio/minio/cmd/generic-handlers.go:600:rateLimit.ServeHTTP() <autogenerated>:1:(*rateLimit).ServeHTTP() /q/.q/sources/gopath/src/github.com/minio/minio/cmd/generic-handlers.go:558:pathValidityHandler.ServeHTTP() <autogenerated>:1:(*pathValidityHandler).ServeHTTP() /q/.q/sources/gopath/src/github.com/minio/minio/cmd/generic-handlers.go:497:httpStatsHandler.ServeHTTP() <autogenerated>:1:(*httpStatsHandler).ServeHTTP() /q/.q/sources/gopath/src/github.com/minio/minio/cmd/generic-handlers.go:65:requestSizeLimitHandler.ServeHTTP() <autogenerated>:1:(*requestSizeLimitHandler).ServeHTTP() /q/.q/sources/gopath/src/github.com/minio/minio/cmd/generic-handlers.go:90:requestHeaderSizeLimitHandler.ServeHTTP() <autogenerated>:1:(*requestHeaderSizeLimitHandler).ServeHTTP() /q/.q/sources/gopath/src/github.com/minio/minio/cmd/crossdomain-xml-handler.go:51:crossDomainPolicy.ServeHTTP() <autogenerated>:1:(*crossDomainPolicy).ServeHTTP() /q/.q/sources/gopath/src/github.com/minio/minio/cmd/generic-handlers.go:219:redirectHandler.ServeHTTP() <autogenerated>:1:(*redirectHandler).ServeHTTP() /q/.q/sources/gopath/src/github.com/minio/minio/cmd/generic-handlers.go:267:minioReservedBucketHandler.ServeHTTP() <autogenerated>:1:(*minioReservedBucketHandler).ServeHTTP() /q/.q/sources/gopath/src/github.com/minio/minio/cmd/generic-handlers.go:246:cacheControlHandler.ServeHTTP() <autogenerated>:1:(*cacheControlHandler).ServeHTTP() /q/.q/sources/gopath/src/github.com/minio/minio/cmd/generic-handlers.go:337:timeValidityHandler.ServeHTTP() <autogenerated>:1:(*timeValidityHandler).ServeHTTP() /q/.q/sources/gopath/src/github.com/minio/minio/vendor/github.com/rs/cors/cors.go:190:(*Cors).Handler.func1() /opt/go/src/net/http/server.go:1918:HandlerFunc.ServeHTTP() /q/.q/sources/gopath/src/github.com/minio/minio/cmd/generic-handlers.go:448:resourceHandler.ServeHTTP() <autogenerated>:1:(*resourceHandler).ServeHTTP() /q/.q/sources/gopath/src/github.com/minio/minio/cmd/auth-handler.go:234:authHandler.ServeHTTP() <autogenerated>:1:(*authHandler).ServeHTTP() /q/.q/sources/gopath/src/github.com/minio/minio/cmd/generic-handlers.go:133:reservedMetadataHandler.ServeHTTP() <autogenerated>:1:(*reservedMetadataHandler).ServeHTTP() /q/.q/sources/gopath/src/github.com/minio/minio/pkg/http/server.go:107:(*Server).Start.func1() /opt/go/src/net/http/server.go:1918:HandlerFunc.ServeHTTP() /opt/go/src/net/http/server.go:2619:serverHandler.ServeHTTP() /opt/go/src/net/http/server.go:1801:(*conn).serve()"
{"client_mode":"local_tempfile","copied_bytes":170,"correlation_id":"01GW9S9JECKT4MBQT4YWWGP40A","filename":"metadata.gz","is_local":true,"is_multipart":false,"is_remote":false,"level":"info","local_temp_path":"/tmp/gitlab","msg":"saved file","remote_id":"","time":"2023-03-24T12:26:15Z"}

So the artifacts are uploaded to minio. I checked and there is plenty of disk space available.

The user is running gitlab-runner (shell) 15.9.1

  • What are you seeing, and how does it differ from what you expect to see?

The runner should not come up with an error and the upload of the artifact should complete without any errors.

  • Consider including screenshots, error messages, and/or other helpful visuals
  • What version are you on (Hint: /help) ? and are you using self-managed or gitlab.com?

We are running self-managed Gitlab 15.9.2 deployed in K8s via Helm Charts

  • What troubleshooting steps have you already taken? Can you link to any docs or other resources so we know where you have been?

Tailed the logs. We had already increased proxy-body-size:

ingress:
    annotations:
        nginx.ingress.kubernetes.io/proxy-body-size: 10g

Any hints are welcome.

Cheers