I have an application that generates approx 100mb of artifacts in a build.
The build executes fine, but then when uploading I pretty regularly get a a few different errors. The runner then retries, and sometimes it goes through, sometimes it doesn’t. My repo is stored on gitlab.com
Here’s an example of a time when it worked on a retry:
Uploading artifacts...
Runtime platform arch=amd64 os=windows pid=1852 revision=a8a019e0 version=12.3.0
builds\XXXX: found 38 matching files
ERROR: Uploading artifacts to coordinator... error error=couldn't execute POST against https://gitlab.com/api/v4/jobs/XXXX/artifacts?artifact_format=zip&artifact_type=archive&expire_in=30+days: Post https://gitlab.com/api/v4/jobs/XXXX/artifacts?artifact_format=zip&artifact_type=archive&expire_in=30+days: write tcp X.X.X.X:50242->35.231.145.151:443: wsasend: An existing connection was forcibly closed by the remote host. id=XXXXX token=XXXX
WARNING: Retrying... error=invalid argument
ERROR: Uploading artifacts to coordinator... error error=couldn't execute POST against https://gitlab.com/api/v4/jobs/XXXX/artifacts?artifact_format=zip&artifact_type=archive&expire_in=30+days: Post https://gitlab.com/api/v4/jobs/XXXX/artifacts?artifact_format=zip&artifact_type=archive&expire_in=30+days: remote error: tls: bad record MAC id=XXXX token=XXXX
WARNING: Retrying... error=invalid argument
Uploading artifacts to coordinator... ok id=XXXX responseStatus=201 Created token=XXXX
Job succeeded
Here’s an example if it failing the entire job because it was unable to perform the upload.
Uploading artifacts...
Runtime platform arch=amd64 os=windows pid=7544 revision=a8a019e0 version=12.3.0
builds\XXXX: found 38 matching files
ERROR: Uploading artifacts to coordinator... error error=couldn't execute POST against https://gitlab.com/api/v4/jobs/XXXX/artifacts?artifact_format=zip&artifact_type=archive&expire_in=30+days: Post https://gitlab.com/api/v4/jobs/XXXX/artifacts?artifact_format=zip&artifact_type=archive&expire_in=30+days: remote error: tls: bad record MAC id=XXXX token=XXXX
WARNING: Retrying... error=invalid argument
ERROR: Uploading artifacts to coordinator... error error=couldn't execute POST against https://gitlab.com/api/v4/jobs/XXXXX/artifacts?artifact_format=zip&artifact_type=archive&expire_in=30+days: Post https://gitlab.com/api/v4/jobs/XXXX/artifacts?artifact_format=zip&artifact_type=archive&expire_in=30+days: read tcp X.X.X.X:50321->35.231.145.151:443: wsarecv: An existing connection was forcibly closed by the remote host. id=XXXX token=XXXX
WARNING: Retrying... error=invalid argument
ERROR: Uploading artifacts to coordinator... error error=couldn't execute POST against https://gitlab.com/api/v4/jobs/XXXX/artifacts?artifact_format=zip&artifact_type=archive&expire_in=30+days: Post https://gitlab.com/api/v4/jobs/XXXX/artifacts?artifact_format=zip&artifact_type=archive&expire_in=30+days: read tcp X.X.X.X:50322->35.231.145.151:443: wsarecv: An existing connection was forcibly closed by the remote host. id=XXXX token=XXXX
FATAL: invalid argument
ERROR: Job failed: Process exited with: 1. Reason was: ()
To workaround this, I’ve configured my job to retry a few times. Typically the build will run again and the artifact upload will succeed. This is a huge pain though, because the builds themselves take approx 1hr. So what should be a 1hr task can pretty easily turn into a 3hr thing.
I’m using a windows host with windows virtual box executors, both running gitlab-runner 12.3.0 (a8a019e0). The machine has a 100mb/s fiber connection. No one else on the same connection is reporting any issues.
Thanks in advance.