Failed to pull image in Kubernetes but succeeded with "docker pull" command with the same credential from gitlab registry

GitLab CI/CD
1

Replace this template with your information

Describe your question in as much detail as possible:
I have docker image in gitlab registry. I login in with “docker login registyr.gitlab.com” and have the credential in my account directory such as ~/.docker/config.json after “docker login” command.

The file looks like below

$ cat ~/.docker/config.json
{
	"auths": {
		"registry.gitlab.com": {
			"auth": "Y3Jxxxxxxxxx="
		}
	},
	"HttpHeaders": {
		"User-Agent": "Docker-Client/19.03.6 (linux)"
	}
}

After login command, I can pull successfully my docker image from gitlab registry. The command I used is

$ docker pull registry.gitlab.com/crong.moim/xxxs/xxx-fileupload

I want to use gitlab regitry with my kubernetes deployment. First I created kubernetes secret with docker login information

$ kubectl create secret generic regcred \
    --from-file=.dockerconfigjson=/home/psw/.docker/config.json \
    --type=kubernetes.io/dockerconfigjson

And I made the deployment yaml for my sample service.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: deployment-xxxxx-fileupload
  namespace: xxxxx-op
spec:
  replicas: 3
  selector:
    matchLabels:
      app: xxxxx-fileupload
  template:
    metadata:
      name: xxxxx-fileupload-pod
      labels:
        app: xxxxx-fileupload
    spec:
      containers:
      - name: container-xxxxx-fileupload
        image: registry.gitlab.com/crong.moim/xxxxx/xxxxx-fileupload:latest
        ports:
        - containerPort: 8003
      imagePullSecrets:
      - name: regcred
---
apiVersion: v1
kind: Service
metadata:
  name: service-xxxxx-fileupload
  namespace: xxxxx-op
spec:
  ports:
    - name: web-port
      port: 8003
      targetPort: 8003
  selector:
    app: xxxxx-fileupload
  type: ClusterIP

However, while applying above deployment, I got errors.

$ kubectl describe pod deployment-xxxxx-fileupload-6cd9496fd6-6cpfn -n xxxxx-op
......
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason     Age               From               Message
  ----     ------     ----              ----               -------
  Normal   Scheduled  20s               default-scheduler  Successfully assigned petus-op/deployment-petus-fileupload-6cd9496fd6-6cpfn to psw-sub
  Normal   BackOff    15s               kubelet, psw-sub   Back-off pulling image "registry.gitlab.com/crong.moim/petus/petus-fileupload:latest"
  Warning  Failed     15s               kubelet, psw-sub   Error: ImagePullBackOff
  Normal   Pulling    3s (x2 over 17s)  kubelet, psw-sub   Pulling image "registry.gitlab.com/crong.moim/petus/petus-fileupload:latest"
  Warning  Failed     2s (x2 over 16s)  kubelet, psw-sub   Failed to pull image "registry.gitlab.com/crong.moim/petus/petus-fileupload:latest": rpc error: code = Unknown desc = Error response from daemon: Get https://registry.gitlab.com/v2/crong.moim/petus/petus-fileupload/manifests/latest: denied: access forbidden
  Warning  Failed     2s (x2 over 16s)  kubelet, psw-sub   Error: ErrImagePull

I DID succeed in pulling image in kubernetes deployment last week on the same server machines. But I re installed kubernetes cluster to study it again on the same machines with the same recipe I used last week.
I don’t know why it failed this time. What shell I do for debug?

I also tried to create the credential with the command below

$ kubectl create secret docker-registry <name> --docker-server=DOCKER_REGISTRY_SERVER --docker-username=DOCKER_USER --docker-password=DOCKER_PASSWORD --docker-email=DOCKER_EMAIL

But I’ve got the errors also.

Thank you.

2

It is absolutely my fault. gitlab is no guilty.

My deployment is in “xxxxx-op” namespace, but I created the registry secret in the “default” namespace. That was the problem.

Gitlab is innocent.

Thanks Gitlab.

1 Like
3

Thanks for pointing this out, I’m stucked for hours until I read this post!