GitLab.com CI/CD artifacts failing to upload but nothing on status page 524 Origin Time-out

violuke · September 13, 2021, 11:40am

Hi,

We are using GitLab CI/CD and getting errors with artifacts being uploaded after a job completes successfully. Nothing has actually changed recently related to this (or the artifacts themselves), but the upload is failing as of today, so I think there is a service issue with GitLab rather than something in our project.

Can this please be investigated?

Thank you,

Luke

taddis · September 13, 2021, 1:10pm

We have the same problem. Sometimes we get 524 Origin Time-out and others 502 Bad Gateway.

Kind regards,
Tobias

violuke · September 13, 2021, 1:53pm

Thanks Tobias, has the issue just recently started for you too?

Our artifact size is just under 100MB of the maximum limit of 1GB. We normally use our own runners installed in Kubernetes, but I get the same issue when using GitLab’s shared runners, so I’m pretty sure this is an issue with GitLab.com

Any word from someone at GitLab?

taddis · September 13, 2021, 2:08pm

Hey Luke,

Our problems started about the same time that you posted the initial post here. We’ve had some other problems with our pipeline since yesterday so I initially thought that it was related to that, but that was not the case.

We have our runner running on GCP (Europe West 3) Compute Engine on Docker.

We haven’t heard anything new yet.

We’re getting three kind of errors at the same stage.

502 Bad Gateway
520 Origin Error
524 Origin Time-out

violuke · September 13, 2021, 2:10pm

Ah, it has appeared on their status page, so it seems that they’re finally aware

https://status.gitlab.com/pages/incident/5b36dc6502d06804c08349f7/613f5ad56e8609053a627730

Thanks for replying so we knew it wasn’t just us

taddis · September 13, 2021, 2:13pm

Ah, nice, thanks for the heads up and for the initial post!

dnsmichi · September 13, 2021, 2:47pm

Thanks for flagging and for your patience. You can follow the incident issue at 2021-09-13: Issues uploading artifacts: 524 or 502 errors (#5521) · Issues · GitLab.com / GitLab Infrastructure Team / production · GitLab and https://twitter.com/gitlabstatus for updates, next to the status page.

DavidKyle · September 13, 2021, 10:13pm

Why was [ 2021-09-13: Issues uploading artifacts: 524 or 502 errors (#5521) · Issues · GitLab.com / GitLab Infrastructure Team / production](2021-09-13: Issues uploading artifacts: 524 or 502 errors (#5521) · Issues · GitLab.com / GitLab Infrastructure Team / production · GitLab) closed?

There are paying customers still seeing this issue.

dnsmichi · September 14, 2021, 11:16am

As far as I can see, the latest update marked the incident issue as resolved.

Apologies for the inconvenience - I would recommend raising your concern directly in the incident issue so that our infrastructure and support team sees your question and can respond accordingly. If you do not want to post publicly, open a support ticket as an alternative please. In this forum topic, it is only me reading at the moment. I will link this topic in the issue though, I forgot about this yesterday.

Cheers,
Michael