I’ve installed GitLab via the helm charts available (gitlab/gitlab) into a k8s cluster.
My CI process involves pushing an image to the GitLab Container Registry, then using kubectl to update the deployment to point to that new image. The pods should pull the image down and run.
Whilst I’m able to push the image up, I’m having trouble getting k8s to pull the image from the registry, the pods fail to pull citing 403 on fetching the manifest. The project is an internal project. Further inspection shows that getting the JWT token itself fails with a 403.
The deployment is set to authenticate using a deployment token (scope read_registry ) for that GitLab project.
The GitLab logs show:
==> /var/log/gitlab/production.log <==
Started GET “/jwt/auth?scope=repository% operations%2Fpersonal-website%3Apull&service=container_registry” for 51.77.118.119 at 2019-08-24 07:15:24 +0000
Processing by JwtController#auth as HTML
Parameters: {“scope”=>“repository: operations/personal-website:pull”, “service”=>“container_registry”}
Completed 403 Forbidden in 11ms (Views: 0.3ms | ActiveRecord: 1.8ms | Elasticsearch: 0.0ms)==> /var/log/gitlab/production_json.log <==
{“method”:“GET”,“path”:“/jwt/auth”,“format”:“html”,“controller”:“JwtController”,“action”:“auth”,“status”:403,“duration”:11.52,“view”:0.28,“db”:1.84,“time”:“2019-08-24T07:15:24.559Z”,“params”:[{“key”:“scope”,“value”:“repository:operations/personal-website:pull”},{“key”:“service”,“value”:“container_registry”}],“remote_ip”:“redacted”,“user_id”:null,“username”:null,“ua”:“docker/18.06.1-ce go/go1.10.3 git-commit/e68fc7a kernel/4.19.0-ovh-xxxx-std-ipv6-64 os/linux arch/amd64 UpstreamClient(Go-http-client/1.1)”,“queue_duration”:3.5,“correlation_id”:“IEsVVHnE1m3”}
The logs from the pods show:
Failed to pull image “registry.:443/operations/personal-website:1924ffb19d29d77d409ef02de6e88f8b71cc87c6”: rpc error: code = Unknown desc = Error response from daemon: Get https://registry.redacted:443/v2/operations/personal-website/manifests/1924ffb19d29d77d409ef02de6e88f8b71cc87c6: denied: access forbidden
Checking the registry logs yields:
time=“2019-08-24T19:48:09.525561465Z” level=warning msg=“error authorizing context: authorization token required” go.version=go1.11.2 http.request.host=“registry.:443” http.request.id=63631e6d-9c4a-4799-b173-c8b9b819fd91 http.request.method=GET http.request.remoteaddr= http.request.uri=“/v2/” http.request.useragent="docker/18.06.1-ce go/go1.10.3 git-commit/e68fc7a kernel/4.19.0-ovh-xxxx-std-ipv6-64 os/linux arch/amd64 UpstreamClient(Go-http-client/1.1)
Note that when I try to use Docker itself to pull from my local computer using the deployment token, it also fails with the same issue.
Any ideas what I’m doing wrong?