I’m trying to upgrade current Gitlab Runner v13.12.0 but anything newer has problems with pulling images to GKE via helm as Back-off pulling image "registry.gitlab.com/gitlab-org/gitlab-runner:alpine-v15.11.0" but when I do Rolling Update to GKE will deploy but then Gitlab Runner fails with error:
ARNING: Failed to pull image with policy "": image pull failed: rpc error: code = Unknown desc = failed to pull and unpack image "registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper:x86_64-436955cb": failed to copy: httpReadSeeker: failed open: unexpected status code https://registry.gitlab.com/v2/gitlab-org/gitlab-runner/gitlab-runner-helper/blobs/sha256:dfebf217e55d1149dc06181e67f12869a1851668b2a3b2186eeb638f01a5ab36: 403 Forbidden
ERROR: Job failed: prepare environment: waiting for pod running: pulling image "registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper:x86_64-436955cb": image pull failed: rpc error: code = Unknown desc = failed to pull and unpack image "registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper:x86_64-436955cb": failed to copy: httpReadSeeker: failed open: unexpected status code https://registry.gitlab.com/v2/gitlab-org/gitlab-runner/gitlab-runner-helper/blobs/sha256:dfebf217e55d1149dc06181e67f12869a1851668b2a3b2186eeb638f01a5ab36: 403 Forbidden. Check https://docs.gitlab.com/runner/shells/index.html#shell-profile-loading for more information
The image you’re trying to deploy does not exist in the registry anymore. OR better said - the layer of gitlab-runner-helper does not exist anymore. Try using another (newer) version of the image.
Actually… since it’s gitlab-runner-helper image, I assume this is not a base image of gitlab-runner image… So try using a newer version of the chart. The currently latest one (as of 15.12.2023) is 0.59.2.
i’ve looked at the helm chart source, and there’s no mention of gitlab-runner-helper, so I can only assume the k8s manifests that are deployed for each job are present in the gitlab-runner source, not in the helm chart.
the repo list you posted is for the helm chart repo. The issue you’re having is with the container. So the container registry would be the right place to look at.
Is there a possibility that your cluster has connectivity issues? Are you using a “private” GKE cluster? Autopilot? Created via TF or via Google console or gcloud CLI? If private, which setup did you use for accessing the internet (and in turn registry.gitlab.com) - cloudrouter / VM with opnsense / VM with iptables DNAT setup / etc…? Are you using VPCs?
Can you try starting a gitlab-runner container by just applying a k8s manifest via kubectl? If yes, can you do the same with the gitlab-runner-helper container? If both work you’ve ruled out the networking issue and there’s something else.