Large Artifact Upload Fails

GitLab CI/CD
1

Howdy folks,

I’ve been trying to upload ~2GB of artifacts to Gitlab, but am getting this error:

ERROR: Uploading artifacts to coordinator… error error=couldn’t execute POST against https://git.onerain.com/api/v4/jobs/25628/artifacts?artifact_format=zip&artifact_type=archive&expire_in=30+days: Post https://git.onerain.com/api/v4/jobs/25628/artifacts?artifact_format=zip&artifact_type=archive&expire_in=30+days: net/http: timeout awaiting response headers id=25628 token=t8k_zdxV
WARNING: Retrying… error=invalid argument
ERROR: Uploading artifacts to coordinator… error error=couldn’t execute POST against https://git.onerain.com/api/v4/jobs/25628/artifacts?artifact_format=zip&artifact_type=archive&expire_in=30+days: Post https://git.onerain.com/api/v4/jobs/25628/artifacts?artifact_format=zip&artifact_type=archive&expire_in=30+days: net/http: timeout awaiting response headers id=25628 token=t8k_zdxV
WARNING: Retrying… error=invalid argument
ERROR: Uploading artifacts to coordinator… error error=couldn’t execute POST against https://git.onerain.com/api/v4/jobs/25628/artifacts?artifact_format=zip&artifact_type=archive&expire_in=30+days: Post https://git.onerain.com/api/v4/jobs/25628/artifacts?artifact_format=zip&artifact_type=archive&expire_in=30+days: net/http: timeout awaiting response headers id=25628 token=t8k_zdxV
FATAL: invalid argument
ERROR: Job failed: exit code 1

Looking in /var/log/gitlab/gitlab-workhorse/current I see:

{“correlation_id”:“aRQJ7UtGjb”,“error”:“handleFileUploads: extract files from multipart: persisting multipart file: unexpected EOF”,“level”:“error”,“method”:“POST”,“msg”:“error”,“time”:“2019-12-04T23:34:56Z”,“uri”:"/api/v4/jobs/25628/artifacts?artifact_format=zip\u0026artifact_type=archive\u0026expire_in=30+days"}

I’ve seen that some other folks have had this problem, and I’ve tried many of the suggestions:

  1. increasing unicorn[‘worker_timeout’]
  2. Made sure artifact maximum size is large enough for instance/group/project
  3. Made sure max attachment size for instance large enough
  4. increase gitlab_workhorse[‘api_ci_long_polling_duration’] = “15m0s” and gitlab_workhorse[‘proxy_headers_timeout’] = “15m0s”
  5. increased nginx[‘proxy_read_timeout’] = 3600 and nginx[‘proxy_connect_timeout’] = 3600
  6. set nginx[‘client_max_body_size’] = 0

I’m on gitlab ce 12.5.2 and gitlab-runner 12.5.0. Our server has plenty of room in the artifacts directory. I’m about out of ideas. Anybody else have any?

Thanks!

2

Hi,

is there an extra HTTP proxy involved? If so, how is it configured?

Cheers,
Michael

3

Nope. The only reverse proxy is the workhorse internal to gitlab

4

Comparing the error messages, I think this might be another case of “Job succeed, but uploading artifacts fails with 500 error” (gitlab-runner#26869).