Hi, I use free private account in gitlab and I have a private project running in a kubernetes cluster. I’m trying to start the service via either a gitlab pipeline using helm chart or manually created kubernetes deployment, but every time i get such an error.
Normal Pulling 11s (x2 over 26s) kubelet Pulling image “registry.gitlab.com/group/project:1.0.0”
Warning Failed 8s (x2 over 24s) kubelet Failed to pull image “registry.gitlab.com/group/project:1.0.0”: rpc error: code = Unknown desc = failed to pull and unpack image “registry.gitlab.com/group/project:1.0.0”: failed to copy: httpReadSeeker: failed open: unexpected status code https://registry.gitlab.com/v2/group/project/blobs/sha256: 403 Forbidden
Warning Failed 8s (x2 over 24s) kubelet Error: ErrImagePull
To pull an image from the private registry, I created a deployment token with all possible scopes and added it to a kubernetes secret with dockerconfigjson inside.
and the deployment is simple
apiVersion: apps/v1 kind: Deployment metadata: labels: app: gateway name: gateway namespace: test spec: replicas: 1 selector: matchLabels: app: gateway template: metadata: labels: app: gateway spec: containers: - image: registry.gitlab.com/group/project:1.0.0 imagePullPolicy: Always name: gateway imagePullSecrets: - name: docker-registry-secret restartPolicy: Always
The issue started yesterday, previously it worker fine. I have an assumption that i reached 10G limit for free account and I cleaned up the half of the space after that. 403 clearly states that kubernetes uses token, but for some reason is still forbidden to pull an image, but if I locally run
docker login registry.gitlab.com -u gitlab+deploy-token -p PASSWORD
I can pull that image. Does anyone have an idea what’s happening?