We have a private Git repository hosted on GitLab.com. We have configured the CI pipelines to use private images hosted on DockerHub and while using the GCE shared runners we are able to run the jobs we have configured. We have then configured a private runner on our EKS cluster in us-west-2 and we are able to run ‘echo hello world’ type jobs with public images.
Unfortunately the combination of GitLab Runner + EKS + private image leads into this error:
ERROR: Job failed: image pull failed: rpc error: code = Unknown desc = Error response from daemon: pull access denied for private/base, repository does not exist or may require ‘docker login’.
We have configured DOCKER_AUTH_CONFIG correctly since the pipelines are able to pull from the shared runners.
Some of the things we tried explicitly.
In .gitlab-ci.yml:
hello alpine:
image: alpine
script:
- echo hello alpine
hello private:
image: "private/base:latest"
script:
- echo hello private repo
hello hub:
image: "registry.hub.docker.com/private/base:latest"
script:
- echo hello hub
And for the DOCKER_AUTH_CONFIG variable under the CI/CD settings:
{
"auths": {
"https://index.docker.io/v1/": {
"auth": "xxxxxxxxxxxxx"
},
"registry.hub.docker.com": {
"auth": "xxxxxxxxxxxxx"
}
}
}
With private being our actual private repository name.
The alpine job runs fine, the ones with private/base or registry.hub.docker.com/private/base just do not work at all and give us the errors:
Running with gitlab-runner 11.10.1 (1f513601)
on Kubernetes runners Z1Fs87ZN
Using Kubernetes namespace: gitlab-managed-apps
Using Kubernetes executor with image registry.hub.docker.com/private/base:latest ...
Waiting for pod gitlab-managed-apps/runner-z1fs87zn-project-12110245-concurrent-3jjx2f to be running, status is Pending
ERROR: Job failed: image pull failed: rpc error: code = Unknown desc = Error response from daemon: pull access denied for registry.hub.docker.com/private/base, repository does not exist or may require 'docker login'
Again on the shared runners with the exact same settings all 3 jobs are completely fine:
Running with gitlab-runner 11.11.0-rc2 (7f58b1ec)
on docker-auto-scale 0277ea0f
Using Docker executor with image registry.hub.docker.com/private/base:latest ...
Pulling docker image registry.hub.docker.com/private/base:latest ...
Using docker image sha256:aaf9aaaa86b6177011d2f5825778658a0aaaad601e48bb54b4aaa25b067faaaa for registry.hub.docker.com/private/base:latest ...
Running on runner-0277ea0f-project-12110245-concurrent-0 via runner-0277ea0f-srm-1557526255-0d27ad05...
This is with EKS 1.12, gitlab cloud and Gitlab Runner 11.10.1. We have other jobs running in this EKS cluster that are not related to GitLab that can pull the private images without any issue, and since the same configuration works on the shared runners I really suspect an issue with Gitlab Runner specifically under EKS.