Docker using registry cache from other repository without access?

GitLab CI/CD
, ,
1

Hi all,

Not sure if this is a Docker issue or a GitLab CI/CD issue, but my builds fail on ‘authorization failed’. What seems to happen is that Docker trying to use an existing image layer from an other repository (in the same group) because it has some overlap, notice #12 and #13 of the build process. However the CI_JOB_TOKEN doesn’t have access to this repository, and it shouldn’t have access. How can I tell Docker not to use layers from other repositories, and shouldn’t it ignore it in the first place, since it doesn’t have access?

This is my build process, DOCKER_TAG = ‘latest’

build:
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
  stage: build
  cache: []
  script:
    - echo ${CI_REGISTRY_IMAGE}
    - echo ${CI_REGISTRY}
    - echo ${CI_JOB_TOKEN} | docker login -u ${CI_REGISTRY_USER} --password-stdin ${CI_REGISTRY}
    - docker buildx ls
    - docker buildx create --use --name multi-arch-builder --driver-opt network=host --driver-opt image=moby/buildkit:v0.10.6
    - docker buildx build --progress=plain --platform=linux/amd64 --pull --push --no-cache --tag="${CI_REGISTRY_IMAGE}/nginx:${DOCKER_TAG}" -f ./docker/nginx/Dockerfile .

This is the output from the GitLab runner:

$ echo ${CI_JOB_TOKEN} | docker login -u ${CI_REGISTRY_USER} --password-stdin ${CI_REGISTRY}
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store
Login Succeeded
$ docker buildx ls
NAME/NODE DRIVER/ENDPOINT STATUS  BUILDKIT                              PLATFORMS
default * docker                                                        
  default default         running v0.11.7-0.20230525183624-798ad6b0ce9f linux/amd64, linux/amd64/v2, linux/amd64/v3, linux/amd64/v4, linux/386
$ docker buildx create --use --name multi-arch-builder --driver-opt network=host --driver-opt image=moby/buildkit:v0.10.6
multi-arch-builder
$ docker buildx build --progress=plain --platform=linux/amd64 --pull --push --no-cache --tag="${CI_REGISTRY_IMAGE}/nginx:${DOCKER_TAG}" -f ./docker/nginx/Dockerfile .
#1 [internal] load .dockerignore
#1 transferring context: 2B done
#1 DONE 0.0s
#2 [internal] load build definition from Dockerfile
#2 transferring dockerfile: 212B done
#2 DONE 0.0s
#3 [internal] load metadata for docker.io/library/nginx:1.24.0-alpine
#3 DONE 0.7s
#4 [1/5] FROM docker.io/library/nginx:1.24.0-alpine@sha256:5e1ccef1e821253829e415ac1e3eafe46920aab0bf67e0fe8a104c57dbfffdf7
#4 resolve docker.io/library/nginx:1.24.0-alpine@sha256:5e1ccef1e821253829e415ac1e3eafe46920aab0bf67e0fe8a104c57dbfffdf7 done
#4 DONE 0.0s
#5 [2/5] WORKDIR /www
#5 CACHED
#6 [internal] load build context
#6 transferring context: 674.31kB done
#6 DONE 0.0s
#7 [3/5] RUN rm -f /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
#7 DONE 0.0s
#8 [4/5] COPY ./docker/nginx/template/ /etc/nginx/templates/
#8 DONE 0.0s
#9 [5/5] COPY ./src /www
#9 DONE 0.0s
#10 exporting to image
#10 exporting layers 0.0s done
#10 exporting manifest sha256:0a0b76ed38daef1153cd2d6f65c0a630300a8b7a4af6db1e1377df02afe354f4 done
#10 exporting config sha256:b8d7d5653a2f872c425df3049dc3c1982882765a8fe1c5b5e807b49993577a8e done
#10 pushing layers
#10 ...
#11 [auth] namespace/project/nginx:pull,push token for registry.gitlab.com
#11 DONE 0.0s
#10 exporting to image
#10 ...
#12 [auth] namespace/project/nginx:pull,push namespace/other-project/nginx:pull token for registry.gitlab.com
#12 DONE 0.0s
#10 exporting to image
#10 ...
#13 [auth] namespace/project/nginx:pull,push namespace/other-project/nginx:pull token for registry.gitlab.com
#13 DONE 0.0s
#10 exporting to image
#10 pushing layers 2.3s done
#10 ERROR: server message: insufficient_scope: authorization failed
------
 > exporting to image:
------
ERROR: failed to solve: server message: insufficient_scope: authorization failed
Cleaning up project directory and file based variables
00:01
ERROR: Job failed: exit code 1

GitLab Runner config.toml if it’s any use:

concurrent = 6
check_interval = 1
shutdown_timeout = 0

[session_server]
  session_timeout = 1800

[[runners]]
  name = ***
  url = "https://gitlab.com"
  id = ***
  token = ***
  token_obtained_at = ***
  token_expires_at = 0001-01-01T00:00:00Z
  executor = "docker"
  [runners.cache]
    MaxUploadedArchiveSize = 0
  [runners.docker]
    tls_verify = false
    image = "alpine:3.18.2"
    privileged = false
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = true
    volumes = ["/var/run/docker.sock:/var/run/docker.sock", "/cache"]
    shm_size = 0

Thanks for reading,

  • Tim
2

Hey Tim,

I ran into the same issue. Were you able to resolve it?

3

Unfortunately not really, I’ve ended up moving the similar build project to a different namespace, disabled the runner cache and removing all stored images from the runner machine:

docker rm $(docker ps -aq)
docker rmi $(docker images -q)
docker volume rm $(docker volume ls -q)

That worked. But as soon as I moved the project back to the same namespace, the issue started happing again. So kept them separated.

4

Thank you for your response, Tim.

The only workaround that seems to be effective is migrating the project to a different namespace, which suggests this may be a GitLab-related issue. I haven’t identified any cache-related problems despite clearing almost everything (volumes, builders, contexts), but nothing else seems to resolve the issue.

However, migrating projects to another namespace is too invasive for us. Could the problem be related to the CI_JOB_TOKEN?