Can't push to container registry

I’ve deployed gitlab to EKS with s3 storage. Most things are working. However, I cannot push effectively to the container registry – I seem to be able to push “a little bit” then docker retries. It does seem to be writing to the s3 bucket, and can see the docker repo in the UI, but the layers don’t end up there and the push eventually errors out.

Any advice or suggestions to debug welcome!

– Shaun


Here is what the push looks like (after successful login):

 % docker push registry.gl.factfiber.com/crane-data/crane-serverless/crane-wisdom:FORLCcK
The push refers to repository [registry.gl.factfiber.com/crane-data/crane-serverless/crane-wisdom]
f3a055bc9a85: Pushing [==================================================>]  3.397MB/3.397MB
b3c9f5bf02a5: Pushing [==================================================>]  1.175MB
3c7b4cd26644: Retrying in 1 second 
1d89cb6aab40: Pushing [==================================================>]  481.1MB/481.1MB
b037ac4b2121: Retrying in 9 seconds 
708d473b8292: Waiting 
52f67b685c25: Waiting 
6e0005c9c743: Waiting 
390e5b4c4fc7: Waiting 
2bd5aa5fa5d1: Waiting 
5280d2327565: Waiting 
77d806cfa004: Waiting 
930c8bc01816: Waiting 
5216338b40a7: Waiting 
received unexpected HTTP status: 500 Internal Server Error

Here is the registry setup in helm values:

      registry:
        storage:
          secret: gitlab-registry-storage
          key: config
        ingress:
          tls:
            secretName: gitlab-registry-tls
          proxyReadTimeout: 900
          proxyBodySize: "0"
          proxyBuffering: "off"

Also global.registry.bucket looks correct. I added the proxy... after googling, but tried originally without with the same effect.

Here is a chunk of log messages from one of the registry pods:

time="2020-05-20T21:02:25.009048293Z" level=error msg="response completed with error" auth.user.name=shauncutts err.code="blob unknown" err.detail=sha256:c8d2203417efc802d4d993d123d67bf65bd9f1fb0c8d2df21f909fd2af0dcdae err.message="blob unknown to registry" go.version=go1.13.9 http.request.host=registry.gl.factfiber.com http.request.id=0ae48c7d-7e41-40bc-9e80-5ff7c3fbb434 http.request.method=HEAD http.request.remoteaddr=10.2.25.173 http.request.uri="/v2/crane-data/crane-serverless/crane-wisdom/blobs/sha256:c8d2203417efc802d4d993d123d67bf65bd9f1fb0c8d2df21f909fd2af0dcdae" http.request.useragent="docker/19.03.8 go/go1.12.17 git-commit/afacb8b kernel/4.19.76-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.8 \(darwin\))" http.response.contenttype="application/json; charset=utf-8" http.response.duration=12.317453ms http.response.status=404 http.response.written=157 vars.digest="sha256:c8d2203417efc802d4d993d123d67bf65bd9f1fb0c8d2df21f909fd2af0dcdae" vars.name="crane-data/crane-serverless/crane-wisdom" 
10.2.4.70 - - [20/May/2020:21:02:24 +0000] "HEAD /v2/crane-data/crane-serverless/crane-wisdom/blobs/sha256:c8d2203417efc802d4d993d123d67bf65bd9f1fb0c8d2df21f909fd2af0dcdae HTTP/1.1" 404 157 "" "docker/19.03.8 go/go1.12.17 git-commit/afacb8b kernel/4.19.76-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.8 \\(darwin\\))"
time="2020-05-20T21:02:25.137214771Z" level=error msg="response completed with error" auth.user.name=shauncutts err.code="blob unknown" err.detail=sha256:7db59e10bf115ede38ae2bfcf88f56c29e79764bc956926e301e06de03339c07 err.message="blob unknown to registry" go.version=go1.13.9 http.request.host=registry.gl.factfiber.com http.request.id=cc9465d4-5e84-4523-8965-e3c960bc6258 http.request.method=HEAD http.request.remoteaddr=10.2.22.68 http.request.uri="/v2/crane-data/crane-serverless/crane-wisdom/blobs/sha256:7db59e10bf115ede38ae2bfcf88f56c29e79764bc956926e301e06de03339c07" http.request.useragent="docker/19.03.8 go/go1.12.17 git-commit/afacb8b kernel/4.19.76-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.8 \(darwin\))" http.response.contenttype="application/json; charset=utf-8" http.response.duration=72.765297ms http.response.status=404 http.response.written=157 vars.digest="sha256:7db59e10bf115ede38ae2bfcf88f56c29e79764bc956926e301e06de03339c07" vars.name="crane-data/crane-serverless/crane-wisdom" 
10.2.4.70 - - [20/May/2020:21:02:25 +0000] "HEAD /v2/crane-data/crane-serverless/crane-wisdom/blobs/sha256:7db59e10bf115ede38ae2bfcf88f56c29e79764bc956926e301e06de03339c07 HTTP/1.1" 404 157 "" "docker/19.03.8 go/go1.12.17 git-commit/afacb8b kernel/4.19.76-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.8 \\(darwin\\))"
10.2.4.70 - - [20/May/2020:21:02:25 +0000] "POST /v2/crane-data/crane-serverless/crane-wisdom/blobs/uploads/ HTTP/1.1" 202 0 "" "docker/19.03.8 go/go1.12.17 git-commit/afacb8b kernel/4.19.76-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.8 \\(darwin\\))"
time="2020-05-20T21:02:25.415237059Z" level=error msg="response completed with error" auth.user.name=shauncutts err.code="blob unknown" err.detail=sha256:e9c911674215196b348508b435ccc23c164192f855040fa792dd480530080a72 err.message="blob unknown to registry" go.version=go1.13.9 http.request.host=registry.gl.factfiber.com http.request.id=2023524c-f144-4c22-ba3a-497787306a59 http.request.method=HEAD http.request.remoteaddr=10.2.26.19 http.request.uri="/v2/crane-data/crane-serverless/crane-wisdom/blobs/sha256:e9c911674215196b348508b435ccc23c164192f855040fa792dd480530080a72" http.request.useragent="docker/19.03.8 go/go1.12.17 git-commit/afacb8b kernel/4.19.76-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.8 \(darwin\))" http.response.contenttype="application/json; charset=utf-8" http.response.duration=14.580756ms http.response.status=404 http.response.written=157 vars.digest="sha256:e9c911674215196b348508b435ccc23c164192f855040fa792dd480530080a72" vars.name="crane-data/crane-serverless/crane-wisdom" 
10.2.4.70 - - [20/May/2020:21:02:25 +0000] "HEAD /v2/crane-data/crane-serverless/crane-wisdom/blobs/sha256:e9c911674215196b348508b435ccc23c164192f855040fa792dd480530080a72 HTTP/1.1" 404 157 "" "docker/19.03.8 go/go1.12.17 git-commit/afacb8b kernel/4.19.76-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.8 \\(darwin\\))"
10.2.4.70 - - [20/May/2020:21:02:25 +0000] "HEAD /v2/crane-data/crane-serverless/crane-wisdom/blobs/sha256:3f01d5292e29c85266665e963ba5c51aaf05595f905d3c6a692859ede31e2ffb HTTP/1.1" 404 157 "" "docker/19.03.8 go/go1.12.17 git-commit/afacb8b kernel/4.19.76-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.8 \\(darwin\\))"
time="2020-05-20T21:02:25.536317543Z" level=error msg="response completed with error" auth.user.name=shauncutts err.code="blob unknown" err.detail=sha256:3f01d5292e29c85266665e963ba5c51aaf05595f905d3c6a692859ede31e2ffb err.message="blob unknown to registry" go.version=go1.13.9 http.request.host=registry.gl.factfiber.com http.request.id=5b8f84b8-3c3d-4709-b7f4-65924f022f7f http.request.method=HEAD http.request.remoteaddr=10.2.26.19 http.request.uri="/v2/crane-data/crane-serverless/crane-wisdom/blobs/sha256:3f01d5292e29c85266665e963ba5c51aaf05595f905d3c6a692859ede31e2ffb" http.request.useragent="docker/19.03.8 go/go1.12.17 git-commit/afacb8b kernel/4.19.76-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.8 \(darwin\))" http.response.contenttype="application/json; charset=utf-8" http.response.duration=9.612996ms http.response.status=404 http.response.written=157 vars.digest="sha256:3f01d5292e29c85266665e963ba5c51aaf05595f905d3c6a692859ede31e2ffb" vars.name="crane-data/crane-serverless/crane-wisdom" 
10.2.4.70 - - [20/May/2020:21:02:25 +0000] "POST /v2/crane-data/crane-serverless/crane-wisdom/blobs/uploads/ HTTP/1.1" 202 0 "" "docker/19.03.8 go/go1.12.17 git-commit/afacb8b kernel/4.19.76-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.8 \\(darwin\\))"
10.2.4.70 - - [20/May/2020:21:02:25 +0000] "HEAD /v2/crane-data/crane-serverless/crane-wisdom/blobs/sha256:c9b1b535fdd91a9855fb7f82348177e5f019329a58c53c47272962dd60f71fc9 HTTP/1.1" 404 157 "" "docker/19.03.8 go/go1.12.17 git-commit/afacb8b kernel/4.19.76-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.8 \\(darwin\\))"
time="2020-05-20T21:02:25.973261885Z" level=error msg="response completed with error" auth.user.name=shauncutts err.code="blob unknown" err.detail=sha256:c9b1b535fdd91a9855fb7f82348177e5f019329a58c53c47272962dd60f71fc9 err.message="blob unknown to registry" go.version=go1.13.9 http.request.host=registry.gl.factfiber.com http.request.id=79da17fe-7851-47b7-a505-a2ef9d69bd6f http.request.method=HEAD http.request.remoteaddr=10.2.26.19 http.request.uri="/v2/crane-data/crane-serverless/crane-wisdom/blobs/sha256:c9b1b535fdd91a9855fb7f82348177e5f019329a58c53c47272962dd60f71fc9" http.request.useragent="docker/19.03.8 go/go1.12.17 git-commit/afacb8b kernel/4.19.76-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.8 \(darwin\))" http.response.contenttype="application/json; charset=utf-8" http.response.duration=20.574985ms http.response.status=404 http.response.written=157 vars.digest="sha256:c9b1b535fdd91a9855fb7f82348177e5f019329a58c53c47272962dd60f71fc9" vars.name="crane-data/crane-serverless/crane-wisdom" 
10.2.4.70 - - [20/May/2020:21:02:26 +0000] "POST /v2/crane-data/crane-serverless/crane-wisdom/blobs/uploads/ HTTP/1.1" 202 0 "" "docker/19.03.8 go/go1.12.17 git-commit/afacb8b kernel/4.19.76-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.8 \\(darwin\\))"
10.2.19.87 - - [20/May/2020:21:04:07 +0000] "GET /v2/crane-data/crane-serverless/tags/list HTTP/1.1" 404 135 "" "GitLab/12.10.6"
time="2020-05-20T21:04:07.481258583Z" level=error msg="response completed with error" auth.user.name= err.code="name unknown" err.detail=map[name:crane-data/crane-serverless] err.message="repository name not known to registry" go.version=go1.13.9 http.request.host="gitlab-registry:5000" http.request.id=7f0e918f-6374-4983-b3e5-ad45f7496ca4 http.request.method=GET http.request.remoteaddr="10.2.19.87:54984" http.request.uri="/v2/crane-data/crane-serverless/tags/list" http.request.useragent="GitLab/12.10.6" http.response.contenttype="application/json; charset=utf-8" http.response.duration=59.547603ms http.response.status=404 http.response.written=135 vars.name="crane-data/crane-serverless" 
time="2020-05-20T21:04:09.54130123Z" level=error msg="response completed with error" auth.user.name= err.code="name unknown" err.detail=map[name:crane-data/crane-serverless/crane-wisdom] err.message="repository name not known to registry" go.version=go1.13.9 http.request.host="gitlab-registry:5000" http.request.id=a08ccc1f-91bb-4e3c-b006-f1a2d47edbaa http.request.method=GET http.request.remoteaddr="10.2.19.87:55002" http.request.uri="/v2/crane-data/crane-serverless/crane-wisdom/tags/list" http.request.useragent="GitLab/12.10.6" http.response.contenttype="application/json; charset=utf-8" http.response.duration=14.863988ms http.response.status=404 http.response.written=148 vars.name="crane-data/crane-serverless/crane-wisdom" 
10.2.19.87 - - [20/May/2020:21:04:09 +0000] "GET /v2/crane-data/crane-serverless/crane-wisdom/tags/list HTTP/1.1" 404 148 "" "GitLab/12.10.6"```

UPDATE -- does it matter that this is a buildkit image?

This is likely an instance of https://docs.gitlab.com/ee/administration/packages/container_registry.html#image-push-errors

However, documentation doesn’t say how to fix it for helm install:

Any help appreciated!