Broken cached layer in registry

How to Use GitLab Self-managed
1

Greetings,

Problem to solve

Our build process fails due to problems with a cached layer. As soon as the build tries to extract the cached layer, we get an unexpected EOF.

We tried to reproduce the error on external infrastructure (store a cached layer on gitlab.com), but the build succeeded. So the Problem has to be in the self-hosted registry.

Configuration

We use Gitlab-ce (18.4) in a container environment (nomad) where the postgres is an own container, s3 buckets store data, the registry is in the same container and a runner is a own vm.

Versions

  • Self-managed

Versions

  • GitLab 18.4.1-ce.0
  • Gitlab-Runner 18.4.0 (docker version 28.4.0/ go1.24.7)
  • Kaniko v1.24.0

Comment

  • We checked the nginx log of the registry
  • We tried external registry for reference
  • The build process has the same issue with local execution (outside of the runner)
  • container push and pull works as expected
  • the cached layer is identical but slightly (<1MB) bigger on the external registry
  • the manifest (docker manifest inspect) of the cached layer is identical (beside the size) in our registry and the external
2

Update:

We figured out, that gitlab in general doesn’t allow downloads of large files.

So far we had a file of around 200MB and the download stopped around 17 s (up to 19 s one time) and around 30 to 80MB (depending on the connection).

Still working on the problem.