Large Artifact Upload Fails

Howdy folks,

I’ve been trying to upload ~2GB of artifacts to Gitlab, but am getting this error:

ERROR: Uploading artifacts to coordinator… error error=couldn’t execute POST against https://git.onerain.com/api/v4/jobs/25628/artifacts?artifact_format=zip&artifact_type=archive&expire_in=30+days: Post https://git.onerain.com/api/v4/jobs/25628/artifacts?artifact_format=zip&artifact_type=archive&expire_in=30+days: net/http: timeout awaiting response headers id=25628 token=t8k_zdxV
WARNING: Retrying… error=invalid argument
ERROR: Uploading artifacts to coordinator… error error=couldn’t execute POST against https://git.onerain.com/api/v4/jobs/25628/artifacts?artifact_format=zip&artifact_type=archive&expire_in=30+days: Post https://git.onerain.com/api/v4/jobs/25628/artifacts?artifact_format=zip&artifact_type=archive&expire_in=30+days: net/http: timeout awaiting response headers id=25628 token=t8k_zdxV
WARNING: Retrying… error=invalid argument
ERROR: Uploading artifacts to coordinator… error error=couldn’t execute POST against https://git.onerain.com/api/v4/jobs/25628/artifacts?artifact_format=zip&artifact_type=archive&expire_in=30+days: Post https://git.onerain.com/api/v4/jobs/25628/artifacts?artifact_format=zip&artifact_type=archive&expire_in=30+days: net/http: timeout awaiting response headers id=25628 token=t8k_zdxV
FATAL: invalid argument
ERROR: Job failed: exit code 1

Looking in /var/log/gitlab/gitlab-workhorse/current I see:

{“correlation_id”:“aRQJ7UtGjb”,“error”:“handleFileUploads: extract files from multipart: persisting multipart file: unexpected EOF”,“level”:“error”,“method”:“POST”,“msg”:“error”,“time”:“2019-12-04T23:34:56Z”,“uri”:"/api/v4/jobs/25628/artifacts?artifact_format=zip\u0026artifact_type=archive\u0026expire_in=30+days"}

I’ve seen that some other folks have had this problem, and I’ve tried many of the suggestions:

  1. increasing unicorn[‘worker_timeout’]
  2. Made sure artifact maximum size is large enough for instance/group/project
  3. Made sure max attachment size for instance large enough
  4. increase gitlab_workhorse[‘api_ci_long_polling_duration’] = “15m0s” and gitlab_workhorse[‘proxy_headers_timeout’] = “15m0s”
  5. increased nginx[‘proxy_read_timeout’] = 3600 and nginx[‘proxy_connect_timeout’] = 3600
  6. set nginx[‘client_max_body_size’] = 0

I’m on gitlab ce 12.5.2 and gitlab-runner 12.5.0. Our server has plenty of room in the artifacts directory. I’m about out of ideas. Anybody else have any?

Thanks!

Hi,

is there an extra HTTP proxy involved? If so, how is it configured?

Cheers,
Michael

Nope. The only reverse proxy is the workhorse internal to gitlab