Runner docker pull fails with insufficient_scope

Problem to solve

We started seeing this intermittent failure with deploy jobs on our self-hosted Gitlab when the runner tries to pull the image that the job is configured to use:

pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed.

The job is configured to pull an image from a different project, and that project has the “Limit access to this project” setting disabled, as most of our deploy jobs across our projects need to access this image. This has been working fine.

It’s odd that this just started happening and that it happens intermittently. Seems like failures due to tokens with invalid scope should happen all the time or never.

Our Gitlab runner is hosted in our EKS cluster and runs jobs in pods on the same cluster.

The only recent change was upgrading from v16.9.3 to v16.10.5 last month.

Would appreciate any help in troubleshooting

Versions

Please select whether options apply, and add the version information.

  • Self-managed
  • GitLab.com SaaS
  • Self-hosted Runners

Versions

  • GitLab: v16.10.5
  • GitLab Runner: v16.10.0

This seems to have been resolved with an upgrade to v16.11.4