Pulling Docker image from GitLab Container Registry stopped working, only for one project

renestalder · June 2, 2022, 3:48pm

I use the GitLab.com free plan to host certain projects for my clients in separate namespaces.

I have some base Docker images hosted in a private repository’s container registry in GitLab. I reference this image from other projects in other namespaces. Furthermore, I’m the sole developer in those namespaces, so It’s clear I have access rights to these docker images on that other repository.

Now, for some reason, I see an error in one project when it tries to pull the docker image:

Failed to pull image with policy "always": Error response from daemon: pull access denied for registry.gitlab.com/<url-to-image>, repository does not exist or may require 'docker login': denied: requested access to the resource is denied (manager.go:203:0s)

This didn’t happen 10 days ago, but happens now. For no particular reason. There was no change regarding the CI setup. And it only seems to happen in that one project. I use it exactly in the same way as 10 days ago, when it worked.

I already tried switching to a different Docker image published in the same container registry as the one failing. But this fails too.

So, the main questions now are

Were there any significant GitLab CI changes since May 23 that might cause this?
Is there any limitation how often a project can pull images from the container registry in GitLab?
Are there any other limitations I might need to know of that could cause the issue?

I have the same user permissions in all projects, in the ones where it still works and the single one, where it doesn’t. I can also pull the image locally.

Any inputs what to look for?

snim2 · July 1, 2022, 12:02pm

Hi @renestalder did you fix this in the end? I’ve had the same problem with images pulled from Docker Hub.

renestalder · July 4, 2022, 8:54am

The only workaround I found was to set the project to public, while restricting all access to the features except the container registry to members only. I found no real solution.

snim2 · July 4, 2022, 8:55am

Oh, that’s interesting, thanks. It hadn’t occurred to me that project visibility might be relevant here.

mkind · July 26, 2022, 9:36pm

Facing the same problem without the possibility to make the project public, I took the following approach:

Create a deploy key with “read registry” permission in the repo for the “source” image
Pull this image in the pipeline of your target repo (docker login/docker pull/docker logout) using the deploy key created above
Now you have this image available locally in your build environment and can access it.

In my case I needed the Docker image located in source repo’s registry as the base image for my Dockerfile in the target repo. The code in the target repo’s pipeline looks basically like this:

  script:
    - docker login -u "$BASE_IMAGE_TOKEN_USER" -p "$BASE_IMAGE_TOKEN_PASSWORD" "$CI_REGISTRY"
    - docker pull "$CI_REGISTRY/my_source_repo:my_source_image_tag"
    - docker logout

BASE_IMAGE_TOKEN_USER and BASE_IMAGE_TOKEN_PASSWORD contain the deploy key created in the source repo stored in CI/CD variables.

tofi86 · August 5, 2022, 10:09pm

I opened a new issue as I also stumpled upon this: "docker pull" with deploy token fails on registry.gitlab.com - but only on public repos which have restricted docker registry to project members (#370039) · Issues · GitLab.org / GitLab · GitLab

perezmeyer · February 17, 2023, 2:43pm

Do you have a link to the issue? I seem to be having the same problem and could not work around it yet.

philippbruhin · February 20, 2023, 9:41am

Same issue here. As mentioned by @renestalder I also had to set the source repo with the docker image Public and restrict access to the features Only Project Members. See following screenshot:

larry834002 · June 6, 2023, 9:54pm

@mkind Your solution works! Here is a my challenge -

When you do a docker pull - where is the image at this point ?
I want to use the above as my base image and add few other stuff on top of this image. docker build and docker are giving errors

Can you please help me , Thanks

oleksiy · July 1, 2023, 12:02am

I solved the problem in Gitlab.com 16.2 via allowing access to my second project CI_JOB_TOKEN

this can be done in your base project in Settings > CI/CD > Token Access
you will need to add full notion of you “client” project, i.e. GROUP/SUBGROUP/PROJECT_NAME

after that in client project use docker login via token

docker login -u $CI_REGISTRY_USER -p $CI_JOB_TOKEN $CI_REGISTRY

oleksiy · July 2, 2023, 3:45pm

It actually works without docker login after granting access
can be used as base image for inherit flows

larry834002 · July 11, 2023, 2:54am

@oleksiy I am also using 16.2. I am not able to make it work. Here are my steps -

Say I have a group/Project where I am maintaining all my images. I used manage and granted access to the other group ( say group B) where I need to access the images. It is giving me errors. However the project ( where I am hosting the images) is showing in shared projects list ( from group B)

Any thoughts as where I am messing it up.

Thanks !!

CordonaCodeCraft · November 27, 2023, 3:05pm

This thread saved my sanity and mental health.

My case was the following.

I am maintaining a repository with common CI/CD assets like jobs shared across other projects and custom images for job runners.

I created a custom image for one of my runners (pre-built with some dependencies).

The image happily pulls locally with no credentials, but It simply failed to pull in a CI/CD context with the following message in the runner’s log:

“ERROR: Job failed: failed to pull image “registry.gitlab.com/redacted-group/redacted-sub-group/redacted-project/sonar-scanner-runner:latest” with specified policies [always]: Error response from daemon: pull access denied for registry.gitlab.com/redacted-group/redacted-sub-group/redacted-project/sonar-scanner-runner:latest repository does not exist or may require ‘docker login’: denied: requested access to the resource is denied (manager.go:237:0s)”

I finally solved this issue at the repository where I pushed the sonar-scanner-runner:latest image.

Go to “Settings” → “CI/CD” → Token Access

In the Token Access section, “Limit access to this project” should be switched on by default as a security measure.

If true, just include all the projects allowed to access the project.

Include all groups and subgroups like this:

parent-group/some-sub-group/a-project-with-access

That’s it.

Many thanks to Oleksiy for helping me out with this.

P.S.

I tried to solve this with ChatGPT, Bard, Claude, and prayers to the eternal Force with a zero result for over 3 hours.

Finally - I settled with the old good way - Googling and digging into forums.

This is a highly kind reminder that a skilled developer is worth 100 more than any LLM.

stevemit · March 19, 2024, 11:12am

@mkind solution works in 16.9 – create a deploy token for the project containing your image.

If you’re trying to reference a pushed image from projects in a group using the Docker executor and an image: entry in .gitlab-ci.yml, put the deploy token into a group variable MY_PROJECT_TOKEN and reference it from another group variable DOCKER_AUTH_CONFIG

/group/target/.gitlab-ci.yml contains:

default:
  # Set GitLab group variables 
  #
  # DOCKER_AUTH_CONFIG (expanded group variable)
  # {"auths":{"gitlab.example.com:5050":{"auth":"$MY_PROJECT_TOKEN"}}}
  #
  # MY_PROJECT_TOKEN (masked group variable)
  # Read-registry permission in project deploy token for /namespace/project
  # EXPIRES after one year.
  # Base64 encoding of username:token
  #
  # https://docs.gitlab.com/ee/user/project/deploy_tokens/
  # https://docs.gitlab.com/ee/ci/docker/using_docker_images.html#access-an-image-from-a-private-container-registry
  image:  gitlab.example.com:5050/namespace/project/my-image