Pushing to container registry gives unsupported status code 500

I’m encountering an issue with pushing images to GitLab Container Registry. It worked just fine previously. Now, it suddenly stops working for just 1 particular image, while the other 13 images in the pipelines are fine. It has been like this for days. The error is:

error pushing image: failed to push to destination registry.gitlab.com/gully-software/navajo/automation:master: unsupported status code 500

I tried removing all related images, built from scratch & pushed but still got the same error.

Additional contexts:

  • I use Kaniko for building & pushing images.
  • Build script for these images are similar, as they are generated from the same template.
  • I’m using a self-hosted GitLab Runner 12.4.1 in autoscaling mode on AWS.

Experiencing the same issue, but only when pushing large images (small images work just fine). The one failing on me is just over 1Gib
Using normal docker build and then docker push on the development machine works just fine.

Is it possible that Kaniko pushes layers in a different way compared to generic Docker engine?

@leminhph do you happen to have any insight to what might be the root cause?

I would also like to note that the registry is behind an NGINX reverse proxy.