Pull access denied for registry.gitlab.com/

I share a gitlab.yml library along with our own runner Docker images across all the projects in my company. It works great, I just need to copy paste some code like

include:
  - project: 'mycompany/operations/gitlab-shared-libraries'
    ref: develop
    file:
      - '/AWS/auth.yml'

aws-auth:
  stage: aws-auth
  extends: .aws-auth
  variables:
    AWS_ACCESS_KEY_ID: $SS_CI_AWS_ACCESS_KEY_ID
    AWS_SECRET_ACCESS_KEY: $SS_CI_AWS_SECRET_ACCESS_KEY
    AWS_DEFAULT_REGION: $PRIMARY_AWS_DEFAULT_REGION

and every pipeline authenticates with AWS and assumes the relevant role for the type of destination environment.

Part of that shared code is this:

.aws-auth:
  image: registry.gitlab.com/mycompany/operations/shared-services/iac-helper:39af1824
  variables:

It works great up, but for some reason, there is this project that keeps on failing because of this:

ERROR: Job failed: failed to pull image "registry.gitlab.com/mycompany/operations/shared-services/iac-helper:39af1824" with specified policies [always]: Error response from daemon: pull access denied for registry.gitlab.com/mycompany/operations/shared-services/iac-helper, repository does not exist or may require 'docker login': denied: requested access to the resource is denied (manager.go:203:0s)

How come every other project runs OK but this one doesn’t? Where can I start looking? this project is part of another group inside the same company, but it can access fine to mycompany/operations/gitlab-shared-libraries I would expect the same with registry.gitlab.com/mycompany/operations/shared-services/iac-helper

I even gave maintainer access to this group over operations but still no luck. Anyone knows where can I continue troubleshooting?

I’m using Gitlab Cloud with free runners (well we buy packages of minutes)

Thanks!

I tried the same thing on a quite similar setup but I couldn’t reproduce your problem. Here is what I tried:

group1/project1 (private) uses an image from group2/project2 registry, and it worked.

Can you share the group hierarchy of your two projects, so I can try to reproduce your scenario.


Otherwise here are some ideas that are worth exploring:

  • It can be a user problem (The used who created the pipeline that pushed the image in the first place)
  • You can use group container registries in GitLab. You can try to push the image to the container registry of a shared parent group and see if the other project can access it.
  • You can create a DOCKER_AUTH_CONFIG (username+ a token) in the accessing project for it to be able to docker login to your regisitry as you.
1 Like

Hello Iduoad,

Thanks for your reply. The group hierarchy is:

data/rango tries to use a Docker image from operations/shared-services registry and it doesn’t work. We moved rango outside the data group and still got permission denied. Any other project outside data can use operations/shared-services registry without a problem.

Just to test, out of desperation, I pushed the Docker image to the registry inside rango and it worked.

Then, I tried to use a shared parent group and I get access denied when pushing a Docker image to a group. It works with repos but it doesn’t allow me to push to any group, but the UI doesn’t give me that option either, so I guess it’s a design principle.

Last, I created a deploy token for operations group and configured DOCKER_AUTH_CONFIG as a CI/CD env var in data group, like you mention and it worked.

But that’s weird, it shouldn’t be like that, this is a startup, we haven’t configured any crazy security yet, everything comes as it was out-of-the-box. Any idea why?

1 Like

Glad the deploy token solution worked for you! I too do think it is not the best and most scalable solution!

Something seems off to me. Given that all the other repos besides rango can access operations/shared-services registry I started suspecting that there is something wrong with the rango repository.

Here is something you can try:

  • Go to Settings>General>Visibility, project features, permissions and compare rango with another repository that has access to operations/shared-services. Also Check the same page for operations/shared-services ans see if there is something weird there.
  • Do the same with members pages. See if you notice something weird.
  • Finally, double-check the failing pipeline trigger-er (if he has the right to pull from the registry). There is a important probability that the problem is related the user running/lingering the pipeline.

Meanwhile, I will try to test scenarios similar to what you have and see if I can reproduce. Also I will try to continue reading the code and see if something pops out


Notes:

Pipeline trigger-er and creds

I went through the code for GitLab-runner and it seems that it first tries DOCKER_AUTH_CONFIG and ~/.docker/config.json or .dockercfg and finally resorts to the Build credentials (which is the case in your scenario).

// ResolveConfigs returns the authentication configuration for docker registries. 
// Goes through several sources in this order: 
// 1. DOCKER_AUTH_CONFIG 
// 2. ~/.docker/config.json or .dockercfg 
// 3. Build credentials 
// Returns a map of registry hostname to RegistryInfo 
func ResolveConfigs(dockerAuthConfig, username string, credentials []common.Credentials, ) (map[string]RegistryInfo, error) 

Here is how the build credentials are defined

module Gitlab
  module Ci
    module Build
      module Credentials
        class Registry < Base
          attr_reader :username, :password

          def initialize(build)
            @username = 'gitlab-ci-token'
            @password = build.token
          end

          def url
            Gitlab.config.registry.host_port
          end

          def valid?
            Gitlab.config.registry.enabled
          end
        end
      end
    end
  end
end

Source code:

Group registries

My bad, the group-level registry is read only, and used for centralized management only.

1 Like

Hello Iduoad,

Thanks for your answer. I checked the repository configuration, I couldn’t see anything. Also checked for some potential inherited DOCKER_AUTH_CONFIG that may be hidden somewhere, but I couldn’t find the issue. We are not catching any files and the repo doesn’t contain any config.json or .dockercfg. We are using Gitlab Cloud and their Shared Runners, so I wouldn’t expect to have any ~/.docker/config.json file hanging around on those.

Lastly, the trigger is always me and I have owner / maintainer privileges across all the repos.
Is there any way to force it to show you what user / types of credentials it’s using?

Thanks,
Ruben

1 Like

Hello Ruben

Sorry for the late reply, I got caught up with work.

I think we are at a dead end here! What makes things harder is that the same setup works on some projects and doesn’t on the other projects.

What I suggest is to open an issue in the GitLab project. and see if they can help you out there.

Not That I know of.

Thanks,
.

1 Like