Gitlab Container Registry: authorization token required when pulling image from EKS Agent

Hi :slight_smile:

We have recently upgraded to Gitlab 15.1, apart of this upgrade required that we upgrade our Kubernetes connection from using the certificate to an agent-based connection.
We are running AWS EKS 1.21

So far we have been able to connect the cluster from the documentation: Sign in · GitLab

When we run a review app pipeline (which is how we previously deployed to the cluster via the certificate method), the pipeline is able to create a deployment in the cluster, but when it tries to pull the image from the gitlab registry it returns the following errors:
GITLAB REGISTRY LOGS

  • “user_agent”:“Go-http-client/2.0”,“written_bytes”:170}
    2022-07-02_08:08:56.65891 time=“2022-07-02T18:08:56.658+10:00” level=warning msg=“error authorizing context: authorization token required” correlation_id=XXX go_version=go1.17.6 root_repo=REPO=“sha256:SHA” vars_name=GITLAB_PROJ version=v3.39.2-gitlab

EKS LOGS

  • Warning Failed 14 minutes kubelet Failed to pull image “registry.symbiote.com.au/dpc/dpc-bandc/jm-review-app:97fbc6b44820e097e7de238f7edee89d718bc878”: rpc error: code = Unknown desc = failed to pull and unpack image “OUR_REGISTRY:IMAGE”: failed to commit snapshot extract- sha256OUR_SHA: open /var/lib/containerd-stargz-grpc/snapshotter/snapshots/1531/fs: too many open files: unknown
  • failed to pull and unpack image

I am wondering if there is something I’m missing with the installation for the agent as the agent isn’t able to authenticate back to the internal gitlab registry to pull the image.

Please let me know if you need more information to help!

Thanks
James

Not sure if you are hitting a bug. Enabling the debug log for the agent could provide more context to analyze.