Kubernetes cannot pull an image from the private registry

Hi, I use free private account in gitlab and I have a private project running in a kubernetes cluster. I’m trying to start the service via either a gitlab pipeline using helm chart or manually created kubernetes deployment, but every time i get such an error.

Normal Pulling 11s (x2 over 26s) kubelet Pulling image “registry.gitlab.com/group/project:1.0.0
Warning Failed 8s (x2 over 24s) kubelet Failed to pull image “registry.gitlab.com/group/project:1.0.0”: rpc error: code = Unknown desc = failed to pull and unpack image “registry.gitlab.com/group/project:1.0.0”: failed to copy: httpReadSeeker: failed open: unexpected status code https://registry.gitlab.com/v2/group/project/blobs/sha256: 403 Forbidden
Warning Failed 8s (x2 over 24s) kubelet Error: ErrImagePull

To pull an image from the private registry, I created a deployment token with all possible scopes and added it to a kubernetes secret with dockerconfigjson inside.

{“auths”:{“registry.gitlab.com”:{“username”:“gitlab+deploy-token”,“password”:“passwor”,“email”:“my_email”,“auth”:“BASE64_AUTH”}}}

and the deployment is simple

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: gateway
  name: gateway
  namespace: test
spec:
  replicas: 1
  selector:
    matchLabels:
      app: gateway
  template:
    metadata:
      labels:
        app: gateway
    spec:
      containers:
      - image: registry.gitlab.com/group/project:1.0.0
        imagePullPolicy: Always
        name: gateway
      imagePullSecrets:
        - name: docker-registry-secret
      restartPolicy: Always

The issue started yesterday, previously it worker fine. I have an assumption that i reached 10G limit for free account and I cleaned up the half of the space after that. 403 clearly states that kubernetes uses token, but for some reason is still forbidden to pull an image, but if I locally run

docker login registry.gitlab.com -u gitlab+deploy-token -p PASSWORD

I can pull that image. Does anyone have an idea what’s happening?

I noticed another weird thing… I was able to run a kubernetes deployment with one particular image tag, but I constantly get 403 Forbidden for other tags

and now i get this

Warning Failed 7s kubelet Failed to pull image “registry.gitlab.com/group/project:1.0.0”: rpc error: code = Unknown desc = failed to pull and unpack image “registry.gitlab.com/group/project:1.0.0”: failed to copy: httpReadSeeker: failed open: failed to do request: Get “https://“registry.gitlab.com/group/project/manifests/sha256:”: dial tcp 35.227.35.254:443: i/o timeout

but from the same ip i can pull one tag and get 403 Forbidden for the other one.
Any help is welcome! it’s completely blocked our deployments for several days already

it seems that the issue was caused by gitlab blocking our IP. Changing the IP fixed the issue

Your problem could have been due to rate limits: GitLab.com settings | GitLab

In particular from that page:

How can I see which IPs are blocked from gitlab ?
And how to change the list ?
I’m using gitlab.com btw, with a private project.

You cannot see, only Gitlab Administrators for gitlab.com can see that. You can open a support ticket with Gitlab about the problems being experienced, but they obviously blocked it for a reason.