Artifacts Failing to Upload (wsasend error)

I have an application that generates approx 100mb of artifacts in a build.

The build executes fine, but then when uploading I pretty regularly get a a few different errors. The runner then retries, and sometimes it goes through, sometimes it doesn’t. My repo is stored on gitlab.com

Here’s an example of a time when it worked on a retry:

Uploading artifacts...
Runtime platform                                    arch=amd64 os=windows pid=1852 revision=a8a019e0 version=12.3.0
builds\XXXX: found 38 matching files          
ERROR: Uploading artifacts to coordinator... error  error=couldn't execute POST against https://gitlab.com/api/v4/jobs/XXXX/artifacts?artifact_format=zip&artifact_type=archive&expire_in=30+days: Post https://gitlab.com/api/v4/jobs/XXXX/artifacts?artifact_format=zip&artifact_type=archive&expire_in=30+days: write tcp X.X.X.X:50242->35.231.145.151:443: wsasend: An existing connection was forcibly closed by the remote host. id=XXXXX token=XXXX
WARNING: Retrying...                                error=invalid argument
ERROR: Uploading artifacts to coordinator... error  error=couldn't execute POST against https://gitlab.com/api/v4/jobs/XXXX/artifacts?artifact_format=zip&artifact_type=archive&expire_in=30+days: Post https://gitlab.com/api/v4/jobs/XXXX/artifacts?artifact_format=zip&artifact_type=archive&expire_in=30+days: remote error: tls: bad record MAC id=XXXX token=XXXX
WARNING: Retrying...                                error=invalid argument
Uploading artifacts to coordinator... ok            id=XXXX responseStatus=201 Created token=XXXX
Job succeeded

Here’s an example if it failing the entire job because it was unable to perform the upload.

Uploading artifacts...
Runtime platform                                    arch=amd64 os=windows pid=7544 revision=a8a019e0 version=12.3.0
builds\XXXX: found 38 matching files          
ERROR: Uploading artifacts to coordinator... error  error=couldn't execute POST against https://gitlab.com/api/v4/jobs/XXXX/artifacts?artifact_format=zip&artifact_type=archive&expire_in=30+days: Post https://gitlab.com/api/v4/jobs/XXXX/artifacts?artifact_format=zip&artifact_type=archive&expire_in=30+days: remote error: tls: bad record MAC id=XXXX token=XXXX
WARNING: Retrying...                                error=invalid argument
ERROR: Uploading artifacts to coordinator... error  error=couldn't execute POST against https://gitlab.com/api/v4/jobs/XXXXX/artifacts?artifact_format=zip&artifact_type=archive&expire_in=30+days: Post https://gitlab.com/api/v4/jobs/XXXX/artifacts?artifact_format=zip&artifact_type=archive&expire_in=30+days: read tcp X.X.X.X:50321->35.231.145.151:443: wsarecv: An existing connection was forcibly closed by the remote host. id=XXXX token=XXXX
WARNING: Retrying...                                error=invalid argument
ERROR: Uploading artifacts to coordinator... error  error=couldn't execute POST against https://gitlab.com/api/v4/jobs/XXXX/artifacts?artifact_format=zip&artifact_type=archive&expire_in=30+days: Post https://gitlab.com/api/v4/jobs/XXXX/artifacts?artifact_format=zip&artifact_type=archive&expire_in=30+days: read tcp X.X.X.X:50322->35.231.145.151:443: wsarecv: An existing connection was forcibly closed by the remote host. id=XXXX token=XXXX
FATAL: invalid argument                            
ERROR: Job failed: Process exited with: 1. Reason was:  ()

To workaround this, I’ve configured my job to retry a few times. Typically the build will run again and the artifact upload will succeed. This is a huge pain though, because the builds themselves take approx 1hr. So what should be a 1hr task can pretty easily turn into a 3hr thing.

I’m using a windows host with windows virtual box executors, both running gitlab-runner 12.3.0 (a8a019e0). The machine has a 100mb/s fiber connection. No one else on the same connection is reporting any issues.

Thanks in advance.

We’re experiencing the same error with a similar situation. Windows runner with an uncompressed artifact set of 327mb as a compressed file it is just over 100mb