I share a gitlab.yml library along with our own runner Docker images across all the projects in my company. It works great, I just need to copy paste some code like
It works great up, but for some reason, there is this project that keeps on failing because of this:
ERROR: Job failed: failed to pull image "registry.gitlab.com/mycompany/operations/shared-services/iac-helper:39af1824" with specified policies [always]: Error response from daemon: pull access denied for registry.gitlab.com/mycompany/operations/shared-services/iac-helper, repository does not exist or may require 'docker login': denied: requested access to the resource is denied (manager.go:203:0s)
How come every other project runs OK but this one doesn’t? Where can I start looking? this project is part of another group inside the same company, but it can access fine to mycompany/operations/gitlab-shared-libraries I would expect the same with registry.gitlab.com/mycompany/operations/shared-services/iac-helper
I even gave maintainer access to this group over operations but still no luck. Anyone knows where can I continue troubleshooting?
I’m using Gitlab Cloud with free runners (well we buy packages of minutes)
I tried the same thing on a quite similar setup but I couldn’t reproduce your problem. Here is what I tried:
group1/project1 (private) uses an image from group2/project2 registry, and it worked.
Can you share the group hierarchy of your two projects, so I can try to reproduce your scenario.
Otherwise here are some ideas that are worth exploring:
It can be a user problem (The used who created the pipeline that pushed the image in the first place)
You can use group container registries in GitLab. You can try to push the image to the container registry of a shared parent group and see if the other project can access it.
You can create a DOCKER_AUTH_CONFIG (username+ a token) in the accessing project for it to be able to docker login to your regisitry as you.
data/rango tries to use a Docker image from operations/shared-services registry and it doesn’t work. We moved rango outside the data group and still got permission denied. Any other project outside data can use operations/shared-services registry without a problem.
Just to test, out of desperation, I pushed the Docker image to the registry inside rango and it worked.
Then, I tried to use a shared parent group and I get access denied when pushing a Docker image to a group. It works with repos but it doesn’t allow me to push to any group, but the UI doesn’t give me that option either, so I guess it’s a design principle.
Last, I created a deploy token for operations group and configured DOCKER_AUTH_CONFIG as a CI/CD env var in data group, like you mention and it worked.
But that’s weird, it shouldn’t be like that, this is a startup, we haven’t configured any crazy security yet, everything comes as it was out-of-the-box. Any idea why?
Glad the deploy token solution worked for you! I too do think it is not the best and most scalable solution!
Something seems off to me. Given that all the other repos besides rango can access operations/shared-services registry I started suspecting that there is something wrong with the rango repository.
Here is something you can try:
Go to Settings>General>Visibility, project features, permissions and compare rango with another repository that has access to operations/shared-services. Also Check the same page for operations/shared-services ans see if there is something weird there.
Do the same with members pages. See if you notice something weird.
Finally, double-check the failing pipeline trigger-er (if he has the right to pull from the registry). There is a important probability that the problem is related the user running/lingering the pipeline.
Meanwhile, I will try to test scenarios similar to what you have and see if I can reproduce. Also I will try to continue reading the code and see if something pops out
Notes:
Pipeline trigger-er and creds
I went through the code for GitLab-runner and it seems that it first tries DOCKER_AUTH_CONFIG and ~/.docker/config.json or .dockercfg and finally resorts to the Build credentials (which is the case in your scenario).
// ResolveConfigs returns the authentication configuration for docker registries.
// Goes through several sources in this order:
// 1. DOCKER_AUTH_CONFIG
// 2. ~/.docker/config.json or .dockercfg
// 3. Build credentials
// Returns a map of registry hostname to RegistryInfo
func ResolveConfigs(dockerAuthConfig, username string, credentials []common.Credentials, ) (map[string]RegistryInfo, error)
Here is how the build credentials are defined
module Gitlab
module Ci
module Build
module Credentials
class Registry < Base
attr_reader :username, :password
def initialize(build)
@username = 'gitlab-ci-token'
@password = build.token
end
def url
Gitlab.config.registry.host_port
end
def valid?
Gitlab.config.registry.enabled
end
end
end
end
end
end
Source code:
Group registries
My bad, the group-level registry is read only, and used for centralized management only.
Thanks for your answer. I checked the repository configuration, I couldn’t see anything. Also checked for some potential inherited DOCKER_AUTH_CONFIG that may be hidden somewhere, but I couldn’t find the issue. We are not catching any files and the repo doesn’t contain any config.json or .dockercfg. We are using Gitlab Cloud and their Shared Runners, so I wouldn’t expect to have any ~/.docker/config.json file hanging around on those.
Lastly, the trigger is always me and I have owner / maintainer privileges across all the repos.
Is there any way to force it to show you what user / types of credentials it’s using?