Hi, I use free private account in gitlab and I have a private project running in a kubernetes cluster. I’m trying to start the service via either a gitlab pipeline using helm chart or manually created kubernetes deployment, but every time i get such an error.
Normal Pulling 11s (x2 over 26s) kubelet Pulling image “registry.gitlab.com/group/project:1.0.0”
Warning Failed 8s (x2 over 24s) kubelet Failed to pull image “registry.gitlab.com/group/project:1.0.0”: rpc error: code = Unknown desc = failed to pull and unpack image “registry.gitlab.com/group/project:1.0.0”: failed to copy: httpReadSeeker: failed open: unexpected status code https://registry.gitlab.com/v2/group/project/blobs/sha256: 403 Forbidden
Warning Failed 8s (x2 over 24s) kubelet Error: ErrImagePull
To pull an image from the private registry, I created a deployment token with all possible scopes and added it to a kubernetes secret with dockerconfigjson inside.
{“auths”:{“registry.gitlab.com”:{“username”:“gitlab+deploy-token”,“password”:“passwor”,“email”:“my_email”,“auth”:“BASE64_AUTH”}}}
and the deployment is simple
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: gateway
name: gateway
namespace: test
spec:
replicas: 1
selector:
matchLabels:
app: gateway
template:
metadata:
labels:
app: gateway
spec:
containers:
- image: registry.gitlab.com/group/project:1.0.0
imagePullPolicy: Always
name: gateway
imagePullSecrets:
- name: docker-registry-secret
restartPolicy: Always
The issue started yesterday, previously it worker fine. I have an assumption that i reached 10G limit for free account and I cleaned up the half of the space after that. 403 clearly states that kubernetes uses token, but for some reason is still forbidden to pull an image, but if I locally run
docker login registry.gitlab.com -u gitlab+deploy-token -p PASSWORD
I can pull that image. Does anyone have an idea what’s happening?