Docker-compose pull command fails with pull access denied

I am using gitlab private registry. As part of the CI/CD job. I login with the CI_REGISTRY_USER as well a token with all the access enabled to gitlab. (tried these two logins instructions separately)

echo $CI_REGISTRY_PASSWORD | docker login -u $CI_REGISTRY_USER --password-stdin $CI_REGISTRY

docker login -u "gitlab-deploy-token" -p "xxxxxxxxxx" $CI_REGISTRY

then start by taking down the currently running service containers and followed by an image pull.

docker-compose -f docker-compose.dev.yaml down --remove-orphans
docker-compose -f docker-compose.dev.yaml --verbose pull
docker-compose -f docker-compose.dev.yaml up -d --no-build --abort-on-container-exit

If I remove the pull instruction everything works correctly. Here is the output of the docker-compose pull. I tried the above instructions with token that is given by Gitlab as well as one created by me (see above). It still doesn’t work. It always gives me pull access denied.

Can someone tell me what is going wrong here ?

docker-compose -f docker-compose.dev.yaml --verbose pull
compose.config.config.find: Using configuration files: ./docker-compose.dev.yaml
docker.utils.config.find_config_file: Trying paths: ['/home/gitlab-runner/.docker/config.json', '/home/gitlab-runner/.dockercfg']
docker.utils.config.find_config_file: Found file at path: /home/gitlab-runner/.docker/config.json
docker.auth.load_config: Found 'auths' section
docker.auth.parse_auth: Found entry (registry='gitlab.xxxx.fi:4567', username='gitlab-deploy-token')
urllib3.connectionpool._make_request: http://localhost:None "GET /v1.38/version HTTP/1.1" 200 865
compose.cli.command.get_client: docker-compose version 1.25.4, build 8d51620a
docker-py version: 4.1.0
CPython version: 3.7.5
OpenSSL version: OpenSSL 1.1.0l  10 Sep 2019
compose.cli.command.get_client: Docker base_url: http+docker://localhost
compose.cli.command.get_client: Docker version: Platform={'Name': 'Docker Engine - Community'}, Components=[{'Name': 'Engine', 'Version': '19.03.6', 'Details': {'ApiVersion': '1.40', 'Arch': 'amd64', 'BuildTime': '2020-02-13T01:26:21.000000000+00:00', 'Experimental': 'false', 'GitCommit': '369ce74a3c', 'GoVersion': 'go1.12.16', 'KernelVersion': '4.15.0-106-generic', 'MinAPIVersion': '1.12', 'Os': 'linux'}}, {'Name': 'containerd', 'Version': '1.2.10', 'Details': {'GitCommit': 'b34a5c8af56e510852c35414db4c1f4fa6172339'}}, {'Name': 'runc', 'Version': '1.0.0-rc8+dev', 'Details': {'GitCommit': '3e425f80a8c931f88e6d94a8c831b9d5aa481657'}}, {'Name': 'docker-init', 'Version': '0.18.0', 'Details': {'GitCommit': 'fec3683'}}], Version=19.03.6, ApiVersion=1.40, MinAPIVersion=1.12, GitCommit=369ce74a3c, GoVersion=go1.12.16, Os=linux, Arch=amd64, KernelVersion=4.15.0-106-generic, BuildTime=2020-02-13T01:26:21.000000000+00:00
compose.cli.verbose_proxy.proxy_callable: docker inspect_network <- ('docker_backend')
urllib3.connectionpool._make_request: http://localhost:None "GET /v1.38/networks/docker_backend HTTP/1.1" 404 52
Pulling postgresql             ... 
Pulling core-application ... 
Pulling frontend         ... 
compose.parallel.feed_queue: Pending: {<Service: postgresql>, <Service: frontend>, <Service: core-application>}
compose.parallel.feed_queue: Starting producer thread for <Service: postgresql>
compose.cli.verbose_proxy.proxy_callable: docker pull <- ('gitlab.xxxx.fi:4567/webapp/custom-postgres', tag='latest', stream=True, platform=None)
docker.auth.get_config_header: Looking for auth config
docker.auth.resolve_authconfig: Looking for auth entry for 'gitlab.xxxx.fi:4567'
docker.auth.resolve_authconfig: Found 'gitlab.xxxx.fi:4567'
docker.auth.get_config_header: Found auth config
compose.parallel.feed_queue: Starting producer thread for <Service: frontend>
compose.cli.verbose_proxy.proxy_callable: docker pull <- ('gitlab.xxxx.fi:4567/webapp/frontend/devops-pipeline-tests', tag='latest', stream=True, platform=None)
compose.parallel.feed_queue: Starting producer thread for <Service: core-application>
docker.auth.get_config_header: Looking for auth config
compose.cli.verbose_proxy.proxy_callable: docker pull <- ('gitlab.xxxx.fi:4567/webapp/core-application/devops-pipeline-tests', tag='latest', stream=True, platform=None)
docker.auth.resolve_authconfig: Looking for auth entry for 'gitlab.xxxx.fi:4567'
docker.auth.get_config_header: Looking for auth config
docker.auth.resolve_authconfig: Found 'gitlab.xxxx.fi:4567'
docker.auth.resolve_authconfig: Looking for auth entry for 'gitlab.xxxx.fi:4567'
docker.auth.get_config_header: Found auth config
docker.auth.resolve_authconfig: Found 'gitlab.xxxx.fi:4567'
docker.auth.get_config_header: Found auth config
compose.parallel.feed_queue: Pending: set()
compose.parallel.feed_queue: Pending: set()
compose.parallel.feed_queue: Pending: set()
compose.parallel.feed_queue: Pending: set()
compose.parallel.feed_queue: Pending: set()
compose.parallel.feed_queue: Pending: set()
compose.parallel.feed_queue: Pending: set()
compose.parallel.feed_queue: Pending: set()
compose.parallel.feed_queue: Pending: set()
compose.parallel.feed_queue: Pending: set()
compose.parallel.feed_queue: Pending: set()
urllib3.connectionpool._make_request: http://localhost:None "POST /v1.38/images/create?tag=latest&fromImage=gitlab.xxxx.fi%3A4567%2Fwebapp%2Fcore-application%2Fdevops-pipeline-tests HTTP/1.1" 404 216
compose.parallel.parallel_execute_iter: Failed: <Service: core-application>
compose.parallel.feed_queue: Pending: set()
urllib3.connectionpool._make_request: http://localhost:None "POST /v1.38/images/create?tag=latest&fromImage=gitlab.xxxx.fi%3A4567%2Fwebapp%2Fcustom-postgres HTTP/1.1" 404 193
compose.parallel.parallel_execute_iter: Failed: <Service: postgresql>
compose.parallel.feed_queue: Pending: set()
urllib3.connectionpool._make_request: http://localhost:None "POST /v1.38/images/create?tag=latest&fromImage=gitlab.xxxx.fi%3A4567%2Fwebapp%2Ffrontend%2Fdevops-pipeline-tests HTTP/1.1" 404 208
compose.parallel.parallel_execute_iter: Failed: <Service: frontend>
compose.parallel.feed_queue: Pending: set()
ERROR: for core-application  pull access denied for gitlab.xxxx.fi:4567/webapp/core-application/devops-pipeline-tests, repository does not exist or may require 'docker login': denied: requested access to the resource is denied
ERROR: for postgresql  pull access denied for gitlab.xxxx.fi:4567/webapp/custom-postgres, repository does not exist or may require 'docker login': denied: requested access to the resource is denied
ERROR: for frontend  pull access denied for gitlab.xxxx.fi:4567/webapp/frontend/devops-pipeline-tests, repository does not exist or may require 'docker login': denied: requested access to the resource is denied
compose.cli.errors.log_api_error: pull access denied for gitlab.xxxx.fi:4567/webapp/frontend/devops-pipeline-tests, repository does not exist or may require 'docker login': denied: requested access to the resource is denied

do you by chance have the docker security enabled? you posted didn’t show the message:
" WARNING! Your password will be stored unencrypted in /home/ubuntu/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store"

what I found is that then I followed those directions and enabled the security stuff, I could login to the registry with out error… But a later push found fail with access denied, like it was not able to retrieve the password from the credential store…

I had to remove the security stuff and now I just delete the file “~/.docker/config.json” when I am done building containers and leave it at that.