GitLab CE registry pull failed with crictl (containerd)

General
1

Hello there,

We have been using GitLab with modern Kubernetes cluster integration, where “containerd” runtime is used. Now we experience issues with pulling images from GitLab registry, both for “crictl” and “ctr”:

# crictl --debug pull env-5903682.madrid.central.jelastic.team:8443/root/gitlab-k8s-cicd-demo/hwservice:master-5
DEBU[0000] get image connection                         
DEBU[0000] connect using endpoint 'unix:///run/containerd/containerd.sock' with '2s' timeout 
DEBU[0000] connected successfully using endpoint: unix:///run/containerd/containerd.sock 
DEBU[0000] PullImageRequest: &PullImageRequest{Image:&ImageSpec{Image:env-5903682.madrid.central.jelastic.team:8443/root/gitlab-k8s-cicd-demo/hwservice:master-5,Annotations:map[string]string{},},Auth:nil,SandboxConfig:nil,} 
DEBU[0000] PullImageResponse: nil                       
FATA[0000] pulling image: rpc error: code = NotFound desc = failed to pull and unpack image "env-5903682.madrid.central.jelastic.team:8443/root/gitlab-k8s-cicd-demo/hwservice:master-5": httpReaderSeeker: failed open: content at https://env-5903682.madrid.central.jelastic.team:8443/v2/root/gitlab-k8s-cicd-demo/hwservice/manifests/sha256:40cabd555bb3457607ccb02ba0c7ca5494a900a293eccae63e68f9929ce2b61d not found: not found

# ctr --debug images pull --skip-verify env-5903682.madrid.central.jelastic.team:8443/root/gitlab-k8s-cicd-demo/hwservice:master-5
DEBU[0000] fetching                                      image="env-5903682.madrid.central.jelastic.team:8443/root/gitlab-k8s-cicd-demo/hwservice:master-5"
DEBU[0000] resolving                                     host="env-5903682.madrid.central.jelastic.team:8443"
DEBU[0000] do request                                    host="env-5903682.madrid.central.jelastic.team:8443" request.header.accept="application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */*" request.header.user-agent=containerd/1.4.6 request.method=HEAD url="https://env-5903682.madrid.central.jelastic.team:8443/v2/root/gitlab-k8s-cicd-demo/hwservice/manifests/master-5"
DEBU[0000] fetch response received                       host="env-5903682.madrid.central.jelastic.team:8443" response.header.content-length=167 response.header.content-type="application/json; charset=utf-8" response.header.date="Fri, 02 Jul 2021 09:15:56 GMT" response.header.docker-distribution-api-version=registry/2.0 response.header.www-authenticate="Bearer realm=\"https://env-5903682.madrid.central.jelastic.team:4848/jwt/auth\",service=\"container_registry\",scope=\"repository:root/gitlab-k8s-cicd-demo/hwservice:pull\"" response.header.x-content-type-options=nosniff response.status="401 Unauthorized" url="https://env-5903682.madrid.central.jelastic.team:8443/v2/root/gitlab-k8s-cicd-demo/hwservice/manifests/master-5"
DEBU[0000] Unauthorized                                  header="Bearer realm=\"https://env-5903682.madrid.central.jelastic.team:4848/jwt/auth\",service=\"container_registry\",scope=\"repository:root/gitlab-k8s-cicd-demo/hwservice:pull\"" host="env-5903682.madrid.central.jelastic.team:8443"
DEBU[0000] do request                                    host="env-5903682.madrid.central.jelastic.team:8443" request.header.accept="application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */*" request.header.user-agent=containerd/1.4.6 request.method=HEAD url="https://env-5903682.madrid.central.jelastic.team:8443/v2/root/gitlab-k8s-cicd-demo/hwservice/manifests/master-5"
DEBU[0000] fetch response received                       host="env-5903682.madrid.central.jelastic.team:8443" response.header.content-length=12400 response.header.content-type=application/vnd.docker.distribution.manifest.v1+prettyjws response.header.date="Fri, 02 Jul 2021 09:15:56 GMT" response.header.docker-content-digest="sha256:40cabd555bb3457607ccb02ba0c7ca5494a900a293eccae63e68f9929ce2b61d" response.header.docker-distribution-api-version=registry/2.0 response.header.etag="\"sha256:40cabd555bb3457607ccb02ba0c7ca5494a900a293eccae63e68f9929ce2b61d\"" response.header.x-content-type-options=nosniff response.status="200 OK" url="https://env-5903682.madrid.central.jelastic.team:8443/v2/root/gitlab-k8s-cicd-demo/hwservice/manifests/master-5"
DEBU[0000] resolved                                      desc.digest="sha256:40cabd555bb3457607ccb02ba0c7ca5494a900a293eccae63e68f9929ce2b61d" host="env-5903682.madrid.central.jelastic.team:8443"
DEBU[0000] fetch schema 1                               
DEBU[0000] do request                                    digest="sha256:40cabd555bb3457607ccb02ba0c7ca5494a900a293eccae63e68f9929ce2b61d" request.header.accept="application/vnd.docker.distribution.manifest.v1+prettyjws, */*" request.header.user-agent=containerd/1.4.6 request.method=GET url="https://env-5903682.madrid.central.jelastic.team:8443/v2/root/gitlab-k8s-cicd-demo/hwservice/manifests/sha256:40cabd555bb3457607ccb02ba0c7ca5494a900a293eccae63e68f9929ce2b61d"
DEBU[0000] fetch response received                       digest="sha256:40cabd555bb3457607ccb02ba0c7ca5494a900a293eccae63e68f9929ce2b61d" response.header.content-length=167 response.header.content-type="application/json; charset=utf-8" response.header.date="Fri, 02 Jul 2021 09:15:56 GMT" response.header.docker-distribution-api-version=registry/2.0 response.header.www-authenticate="Bearer realm=\"https://env-5903682.madrid.central.jelastic.team:4848/jwt/auth\",service=\"container_registry\",scope=\"repository:root/gitlab-k8s-cicd-demo/hwservice:pull\"" response.header.x-content-type-options=nosniff response.status="401 Unauthorized" url="https://env-5903682.madrid.central.jelastic.team:8443/v2/root/gitlab-k8s-cicd-demo/hwservice/manifests/sha256:40cabd555bb3457607ccb02ba0c7ca5494a900a293eccae63e68f9929ce2b61d"
DEBU[0000] Unauthorized                                  digest="sha256:40cabd555bb3457607ccb02ba0c7ca5494a900a293eccae63e68f9929ce2b61d" header="Bearer realm=\"https://env-5903682.madrid.central.jelastic.team:4848/jwt/auth\",service=\"container_registry\",scope=\"repository:root/gitlab-k8s-cicd-demo/hwservice:pull\""
DEBU[0000] do request                                    digest="sha256:40cabd555bb3457607ccb02ba0c7ca5494a900a293eccae63e68f9929ce2b61d" request.header.accept="application/vnd.docker.distribution.manifest.v1+prettyjws, */*" request.header.user-agent=containerd/1.4.6 request.method=GET url="https://env-5903682.madrid.central.jelastic.team:8443/v2/root/gitlab-k8s-cicd-demo/hwservice/manifests/sha256:40cabd555bb3457607ccb02ba0c7ca5494a900a293eccae63e68f9929ce2b61d"
DEBU[0000] fetch response received                       digest="sha256:40cabd555bb3457607ccb02ba0c7ca5494a900a293eccae63e68f9929ce2b61d" response.header.content-length=211 response.header.content-type="application/json; charset=utf-8" response.header.date="Fri, 02 Jul 2021 09:15:57 GMT" response.header.docker-distribution-api-version=registry/2.0 response.header.x-content-type-options=nosniff response.status="404 Not Found" url="https://env-5903682.madrid.central.jelastic.team:8443/v2/root/gitlab-k8s-cicd-demo/hwservice/manifests/sha256:40cabd555bb3457607ccb02ba0c7ca5494a900a293eccae63e68f9929ce2b61d"
ctr: httpReaderSeeker: failed open: content at https://env-5903682.madrid.central.jelastic.team:8443/v2/root/gitlab-k8s-cicd-demo/hwservice/manifests/sha256:40cabd555bb3457607ccb02ba0c7ca5494a900a293eccae63e68f9929ce2b61d not found: not found

At the same time, pulling via “docker” doesn’t cause any issues:

# docker pull env-5903682.madrid.central.jelastic.team:8443/root/gitlab-k8s-cicd-demo/hwservice:master-5
master-5: Pulling from root/gitlab-k8s-cicd-demo/hwservice
df20fa9351a1: Pull complete
60d37dc3360e: Pull complete
bdb33821fbac: Pull complete
c2e82bf56b21: Pull complete
21aadbca6870: Pull complete
f466024ee4b6: Pull complete
Digest: sha256:766f6f13b0c7026003aed6286389aa91524fea989c93fddc1c9270fea08e638c
Status: Downloaded newer image for env-5903682.madrid.central.jelastic.team:8443/root/gitlab-k8s-cicd-demo/hwservice:master-5
env-5903682.madrid.central.jelastic.team:8443/root/gitlab-k8s-cicd-demo/hwservice:master-5

We use a self-signed certificate on GitLab instance — in theory it may be the reason. But, “containerd” and “ctr” have options to ignore the certificate validity, and the error message doesn’t refer it. The Docker client also doesn’t have issues with the self-signed certificate, added to exclusions.

Any help here on how to fix or workaroud the pull issue is highy appreciated.

2

EDIT: after a deeper investigation, we have figured out, that ALL images pulls via “crictl”, “crt” are affected — regardless of a certificate validity. The trusted registry’s image pull printout:

# crictl --debug pull env-6496969.paas.hosted-by-previder.com:5000/root/gitlab-k8s-cicd-demo/hwservice:master-2
DEBU[0000] get image connection                         
DEBU[0000] connect using endpoint 'unix:///run/containerd/containerd.sock' with '2s' timeout 
DEBU[0000] connected successfully using endpoint: unix:///run/containerd/containerd.sock 
DEBU[0000] PullImageRequest: &PullImageRequest{Image:&ImageSpec{Image:env-6496969.paas.hosted-by-previder.com:5000/root/gitlab-k8s-cicd-demo/hwservice:master-2,Annotations:map[string]string{},},Auth:nil,SandboxConfig:nil,} 
DEBU[0000] PullImageResponse: nil                       
FATA[0000] pulling image: rpc error: code = NotFound desc = failed to pull and unpack image "env-6496969.paas.hosted-by-previder.com:5000/root/gitlab-k8s-cicd-demo/hwservice:master-2": httpReaderSeeker: failed open: content at https://env-6496969.paas.hosted-by-previder.com:5000/v2/root/gitlab-k8s-cicd-demo/hwservice/manifests/sha256:7fa792f48c296ba038d394487dc55dbe71e996404e2e683cc778cbe5824fcf2a not found: not found

# ctr --debug images pull env-6496969.paas.hosted-by-previder.com:5000/root/gitlab-k8s-cicd-demo/hwservice:master-2
DEBU[0000] fetching                                      image="env-6496969.paas.hosted-by-previder.com:5000/root/gitlab-k8s-cicd-demo/hwservice:master-2"
DEBU[0000] resolving                                     host="env-6496969.paas.hosted-by-previder.com:5000"
DEBU[0000] do request                                    host="env-6496969.paas.hosted-by-previder.com:5000" request.header.accept="application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */*" request.header.user-agent=containerd/1.3.7 request.method=HEAD url="https://env-6496969.paas.hosted-by-previder.com:5000/v2/root/gitlab-k8s-cicd-demo/hwservice/manifests/master-2"
DEBU[0000] fetch response received                       host="env-6496969.paas.hosted-by-previder.com:5000" response.header.content-length=167 response.header.content-type="application/json; charset=utf-8" response.header.date="Tue, 06 Jul 2021 12:33:49 GMT" response.header.docker-distribution-api-version=registry/2.0 response.header.www-authenticate="Bearer realm=\"https://env-6496969.paas.hosted-by-previder.com:443/jwt/auth\",service=\"container_registry\",scope=\"repository:root/gitlab-k8s-cicd-demo/hwservice:pull\"" response.header.x-content-type-options=nosniff response.status="401 Unauthorized" url="https://env-6496969.paas.hosted-by-previder.com:5000/v2/root/gitlab-k8s-cicd-demo/hwservice/manifests/master-2"
DEBU[0000] Unauthorized                                  header="Bearer realm=\"https://env-6496969.paas.hosted-by-previder.com:443/jwt/auth\",service=\"container_registry\",scope=\"repository:root/gitlab-k8s-cicd-demo/hwservice:pull\"" host="env-6496969.paas.hosted-by-previder.com:5000"
DEBU[0000] do request                                    host="env-6496969.paas.hosted-by-previder.com:5000" request.header.accept="application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */*" request.header.user-agent=containerd/1.3.7 request.method=HEAD url="https://env-6496969.paas.hosted-by-previder.com:5000/v2/root/gitlab-k8s-cicd-demo/hwservice/manifests/master-2"
DEBU[0000] fetch response received                       host="env-6496969.paas.hosted-by-previder.com:5000" response.header.content-length=12400 response.header.content-type=application/vnd.docker.distribution.manifest.v1+prettyjws response.header.date="Tue, 06 Jul 2021 12:33:49 GMT" response.header.docker-content-digest="sha256:7fa792f48c296ba038d394487dc55dbe71e996404e2e683cc778cbe5824fcf2a" response.header.docker-distribution-api-version=registry/2.0 response.header.etag="\"sha256:7fa792f48c296ba038d394487dc55dbe71e996404e2e683cc778cbe5824fcf2a\"" response.header.x-content-type-options=nosniff response.status="200 OK" url="https://env-6496969.paas.hosted-by-previder.com:5000/v2/root/gitlab-k8s-cicd-demo/hwservice/manifests/master-2"
DEBU[0000] resolved                                      desc.digest="sha256:7fa792f48c296ba038d394487dc55dbe71e996404e2e683cc778cbe5824fcf2a" host="env-6496969.paas.hosted-by-previder.com:5000"
DEBU[0000] fetch schema 1                               
DEBU[0000] do request                                    digest="sha256:7fa792f48c296ba038d394487dc55dbe71e996404e2e683cc778cbe5824fcf2a" request.header.accept="application/vnd.docker.distribution.manifest.v1+prettyjws, */*" request.header.user-agent=containerd/1.3.7 request.method=GET url="https://env-6496969.paas.hosted-by-previder.com:5000/v2/root/gitlab-k8s-cicd-demo/hwservice/manifests/sha256:7fa792f48c296ba038d394487dc55dbe71e996404e2e683cc778cbe5824fcf2a"
DEBU[0000] fetch response received                       digest="sha256:7fa792f48c296ba038d394487dc55dbe71e996404e2e683cc778cbe5824fcf2a" response.header.content-length=211 response.header.content-type="application/json; charset=utf-8" response.header.date="Tue, 06 Jul 2021 12:33:49 GMT" response.header.docker-distribution-api-version=registry/2.0 response.header.x-content-type-options=nosniff response.status="404 Not Found" url="https://env-6496969.paas.hosted-by-previder.com:5000/v2/root/gitlab-k8s-cicd-demo/hwservice/manifests/sha256:7fa792f48c296ba038d394487dc55dbe71e996404e2e683cc778cbe5824fcf2a"
ctr: httpReaderSeeker: failed open: content at https://env-6496969.paas.hosted-by-previder.com:5000/v2/root/gitlab-k8s-cicd-demo/hwservice/manifests/sha256:7fa792f48c296ba038d394487dc55dbe71e996404e2e683cc778cbe5824fcf2a not found: not found
3

Containerd developers pointed out that it should be the registry issue.
Wondered, if nobody stumbled upon this issue before.

4

Related issues:

5

Katrin, thanks for the response. Although, we have issues with pulling images from the container registry to e.g. containerd-driven Kubernetes node. It’s not related to the runners usage.

6

I’ve replied in parallel to your ZenDesk ticket :wink:

7

Ok, I see. The last issue mentioned covers the pull “by-digest”, but we don’t use it. We’re trying to get an ordinary image pull working, as described in the containerd ticket, or the example above.
The containerd team stated, that it could be issue in the registry backend.

8

FYI: GitLab-CE registry pull failed with containerd (#337184) · Issues · GitLab.org / GitLab · GitLab

1 Like