GitLab DinD Runner - Docker Login leads to "denied: access forbidden"

Hello,

after two days of searching for a working configuration by myself I now decided to ask for help here. Some time ago I installed GitLab on my Debian Jessie server by using the gitlab/gitlab-ce:latest Docker image. I installed a working configuration of two GitLab runners by the gitlab/gitlab-runner:latest Docker image. One of them is specialized on building Docker images:

concurrent = 4
check_interval = 0

[[runners]]
  name = "Gitlab-Runner-Docker"
  url = "[Anonymous]"
  token = "[Anonymous]"
  executor = "docker"
  limit = 4
  privileged = true
  cache_dir = "cache"
  [runners.docker]
    tls_verify = false
    image = "docker:latest"
    privileged = true
    disable_cache = false
    volumes = ["/cache"]
    shm_size = 0
    cpuset_cpus = "4-7"
    allowed_images = ["docker:latest"]
  [runners.cache]

This runner is able to build images and push them to Docker Hub without any problem (The runner is tagged “docker”):

backend_deploy_master:
    stage: deploy
    tags:
        - docker
    only:
        - master@[Anonymous]
    services:
        - docker:dind
    environment:
        name: backenend_deployment_master
        url: [Anonymous]
    script:
        - docker build -t [Anonymous] .
        - docker login -u [Anonymous] -p [Anonymous]
        - docker push [Anonymous]

As I activated the Docker Container Registry, too, I now tried to push images to my local Registry instead of to Docker Hub. So i modified the above code to use my repository instead:

backend_deploy_master:
    stage: deploy
    tags:
        - docker
    only:
        - master@[Anonymous]
    services:
        - docker:dind
    environment:
        name: backenend_deployment_master
        url: [Anonymous]
    script:
        - docker build -t [Anonymous] .
        - docker login -u [Anonymous] -p [Anonymous] docker.gitlab.[Anonymous].de
        - docker push [Anonymous]

This fails at the docker login command with this error message:

WARNING! Using --password via the CLI is insecure. Use --password-stdin.
Error response from daemon: Get https://docker.gitlab.[Anonymous].de/v2/: denied: access forbidden

My first thought was that my Docker internal registry is faulty, so I executed docker login -u [Anonymous] -p [Anonymous] docker.gitlab.[Anonymous].de on my host machine. Result was Login Succeeded and even pushing images was exactly no problem. Just to be sure I tried docker login -u [Anonymous] -p [Anonymous] docker.gitlab.[Anonymous].de on several other Linux machines, too. It always succeeded. Of course the [Anonymous] data was exactly the same all the time.

I decided to take a look at the registry logs for this. This is the log output if I run docker login on my host:

2018-08-22_10:34:15.07620 time="2018-08-22T10:34:15.076167544Z" level=warning msg="error authorizing context: authorization token required" environment=production go.version=go1.10.3 http.request.host=docker.gitlab.[Anonymous].de http.request.id=b33ea6bc-17ae-4a26-b163-0e943b844f5a http.request.method=GET http.request.remoteaddr=[Anonymous] http.request.uri="/v2/" http.request.useragent="docker/17.06.0-ce go/go1.8.3 git-commit/02c1d87 kernel/3.16.0-4-amd64 os/linux arch/amd64 UpstreamClient(Docker-Client/17.06.0-ce \\(linux\\))" instance.id=3c80e5c7-932a-417b-b78e-9e8d0604fa07 service=registry version=v2.6.2-2-g91c17ef 
2018-08-22_10:34:15.07621 127.0.0.1 - - [22/Aug/2018:10:34:15 +0000] "GET /v2/ HTTP/1.1" 401 87 "" "docker/17.06.0-ce go/go1.8.3 git-commit/02c1d87 kernel/3.16.0-4-amd64 os/linux arch/amd64 UpstreamClient(Docker-Client/17.06.0-ce \\(linux\\))"
2018-08-22_10:34:15.21670 time="2018-08-22T10:34:15.21667828Z" level=info msg="response completed" environment=production go.version=go1.10.3 http.request.host=docker.gitlab.[Anonymous].de http.request.id=c28d24b5-8532-4052-910e-1e7ef99c987e http.request.method=GET http.request.remoteaddr=[Anonymous] http.request.uri="/v2/" http.request.useragent="docker/17.06.0-ce go/go1.8.3 git-commit/02c1d87 kernel/3.16.0-4-amd64 os/linux arch/amd64 UpstreamClient(Docker-Client/17.06.0-ce \\(linux\\))" http.response.contenttype="application/json; charset=utf-8" http.response.duration=1.23239ms http.response.status=200 http.response.written=2 instance.id=3c80e5c7-932a-417b-b78e-9e8d0604fa07 service=registry version=v2.6.2-2-g91c17ef 
2018-08-22_10:34:15.21673 127.0.0.1 - - [22/Aug/2018:10:34:15 +0000] "GET /v2/ HTTP/1.1" 200 2 "" "docker/17.06.0-ce go/go1.8.3 git-commit/02c1d87 kernel/3.16.0-4-amd64 os/linux arch/amd64 UpstreamClient(Docker-Client/17.06.0-ce \\(linux\\))"

This was the output if the GitLab runner tries to login:

2018-08-22_10:31:35.41572 time="2018-08-22T10:31:35.415677686Z" level=warning msg="error authorizing context: authorization token required" environment=production go.version=go1.10.3 http.request.host=docker.gitlab.[Anonymous].de http.request.id=e40f6776-e982-4a41-876e-501904e8c067 http.request.method=GET http.request.remoteaddr=172.17.0.8 http.request.uri="/v2/" http.request.useragent="docker/18.06.0-ce go/go1.10.3 git-commit/0ffa825 kernel/3.16.0-4-amd64 os/linux arch/amd64 UpstreamClient(Docker-Client/18.06.0-ce \\(linux\\))" instance.id=3c80e5c7-932a-417b-b78e-9e8d0604fa07 service=registry version=v2.6.2-2-g91c17ef 
2018-08-22_10:31:35.41574 127.0.0.1 - - [22/Aug/2018:10:31:35 +0000] "GET /v2/ HTTP/1.1" 401 87 "" "docker/18.06.0-ce go/go1.10.3 git-commit/0ffa825 kernel/3.16.0-4-amd64 os/linux arch/amd64 UpstreamClient(Docker-Client/18.06.0-ce \\(linux\\))"

It seems like the GitLab runner fails to execute the steps 3-5 described in this document https://docs.gitlab.com/omnibus/architecture/registry/, but I have no idea why. Every other client I tried is able to login.

As I am not sure whether I configured something wrong I will paste the registry parts of my gitlab.rb in here, too:

################################################################################
## Container Registry settings
##! Docs: https://docs.gitlab.com/ce/administration/container_registry.html
################################################################################

registry_external_url 'https://docker.gitlab.[Anonymous].de'

### Settings used by GitLab application
gitlab_rails['registry_enabled'] = true
gitlab_rails['registry_host'] = "gitlab.docker.[Anonymous].de"
gitlab_rails['registry_port'] = "80"
gitlab_rails['registry_path'] = "/var/opt/gitlab/gitlab-rails/shared/registry"
################################################################################
## Registry NGINX
################################################################################

registry_nginx['enable'] = true
registry_nginx['listen_port'] = 5226 
registry_nginx['listen_https'] = false
registry_nginx['proxy_set_headers'] = {
 "Host" => "$http_host",
 "X-Real-IP" => "$remote_addr",
 "X-Forwarded-For" => "$proxy_add_x_forwarded_for",
 "X-Forwarded-Proto" => "https",
 "X-Forwarded-Ssl" => "on"
}

The main GitLab and the Registry run on the same Docker container started with:

docker run --detach \
    --hostname gitlab.[Anonymous].de \
    --publish 5223:443 \
    --publish 5224:80 \
    --publish 5225:22 \
    --publish 5226:5226 \
    --publish 5227:5227 \
    --name gitlab-ce \
    --restart always \
    --volume ~/gitlab_data/config:/etc/gitlab \
    --volume ~/gitlab_data/logs:/var/log/gitlab \
    --volume ~/gitlab_data/data:/var/opt/gitlab \
    gitlab/gitlab-ce:latest

The GitLab runner is started with:

docker run -d --name gitlab-runner-docker --restart always \
  -v /home/docker/gitlab-runner/gitlab-runner-docker:/etc/gitlab-runner \
  -v /var/run/docker.sock:/var/run/docker.sock \
  gitlab/gitlab-runner:latest

Both, the main GitLab and the Registry run behind a Apache proxy encrypted with LetsEncrypt:

For gitlab.[Anonymous].de (Main GitLab):

ProxyPreserveHost On

ProxyPassMatch ^\/.well-known\/acme-challenge\/[abcdefghijklmnopqrstuvwxyz\/\-._ABCDEFGHIJKLMNOPQRSTUVWXYZ1234567890]*$ !
ProxyPass / http://localhost:5224/
ProxyPassReverse / http://localhost:5224/

For docker.gitlab.[Anonymous].de (Registry):

ProxyPreserveHost On

ProxyPassMatch ^\/.well-known\/acme-challenge\/[abcdefghijklmnopqrstuvwxyz\/\-._ABCDEFGHIJKLMNOPQRSTUVWXYZ1234567890]*$ !
ProxyPass / http://localhost:5226/
ProxyPassReverse / http://localhost:5226/

Does anyone of you have any idea what I might want to change to get this running? Thank you in advance.

hello,
Can you reslove this problem? I also met the same problem.

Hello, I have the same issue. Any idea on how to resolve it ? thanks