Kubernetes cannot pull images for deployments installed by gitlab-agent

Infrastructure as Code & Cloud Native
, ,
1

Hello !

I manage a kubernetes cluster where we deploy different projects privately stored on gitlab.com using the gitlab Auto DevOps template (via a custom pipeline definition, not via the toggle found in the project’s CI/CD settings). We have setup a gitlab agent to connect our pipelines and the cluster.

Deploying our projects works really well, but when our deployments are moved to a different cluster node than the one they were installed on by the agent the image cannot be pulled successfully from our gitlab container registry with the following error:

Failed to pull image "registry.gitlab.com/xxxx/xxxx/dev:5f5440fb44e2702993525990f1c90f9f865fee72":
failed to pull and unpack image "registry.gitlab.com/xxxx/xxxx/dev:5f5440fb44e2702993525990f1c90f9f865fee72":
failed to resolve reference "registry.gitlab.com/xxxx/xxxx/dev:5f5440fb44e2702993525990f1c90f9f865fee72":
failed to authorize: failed to fetch oauth token: unexpected status from GET request to https://gitlab.com/jwt/auth?scope=repository%3Axxxx%2Fxxxx%2Fdev%3Apull&service=container_registry: 401 Unauthorized

I assume it is because the secret used to initially pull the image has expired as this doesn’t seem to happen if the deployment was created less than 24h prior to moving it to another node.

I tried to find a way to specify a deploy token as an imagePullSecret but could not find a way to make it work with Auto DevOps.

Is there a way to make sure our deployed apps can still be pulled long after being installed by the gitlab agent ?

Versions:

  • Gitlab agent for Kubernetes: 18.3.0
  • Gitlab: latest on gitlab.com
  • Kubernetes: 1.32.7

Thanks in advance !

2

In case anyone comes across this issue aswell, turns out that if a deploy token named ‘gitlab-deploy-token’ is created, gitlab will automatically make it available through the ‘CI_DEPLOY_USER’ and a ‘CI_DEPLOY_PASSWORD’ CI/CD variables. It is undocumented, but the gitlab auto-devops template will automatically use those variables to create the imagePullSecret if available. This allowed us to create an unlimited deploy token and ensure our images can be re-pulled long after the initial deployment.

To make sure this deploy token is available in all nested projects in your group, you’ll have to manually create a ‘CI_DEPLOY_USER’ and a ‘CI_DEPLOY_PASSWORD’ CI/CD variables in the group as gitlab won’t auto-populate them in projects nested in sub-groups.