Docker Machine docker-in-docker "Cannot connect to the Docker daemon at tcp://localhost:2375/. Is the docker daemon running?"

Hi community,
I am attempting to set up docker machine runners on my vsphere environment. Docker machine runner starts a VM but then the docker container to be executed by docker compose fails with an error:

Cannot connect to the Docker daemon at tcp://localhost:2375/. Is the docker daemon running? I have also tried this with tcp://docker:2375 and in doing this it attempts to resolve this in DNS and connect via the squid proxy servicing my environment and fails.

I have followed various guides and forum posts to come to the configuration I have with various combinations of settings in my .gitlab-ci.yml file such as:

image: docker:dind
image: docker
variables:
    DOCKER_HOST: tcp://localhost:2375/
    DOCKER_TLS_CERTDIR: ""

or

image: docker   (also tried this and docker:dind)
variables:
    DOCKER_HOST: tcp://docker:2375/     also tried localhost in here.
    DOCKER_DRIVER: overlay2
    DOCKER_TLS_CERTDIR: ""
services:
    - name: docker:dind
      alias: docker
      command: ["--tls=false"]

In my runner I have config:

  [runners.docker]
    tls_verify = false
    image = "alpine:latest"
    privileged = true
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = false
    volumes = ["/cache", "/var/run/docker.sock:/var/run/docker.sock"]
    shm_size = 0
  [runners.machine]
    IdleCount = 0
    IdleTime = 60
    IdleScaleFactor = 0.0
    IdleCountMin = 0
    MachineDriver = "vmwarevsphere"

So privileged is set true and a volume is mounted allowing dind containers access to the docker socket.

With all my attempts I get one of two results:-

  1. Docker login succeeds in the .gitlab-ci.yml in the before_script but when executing docker compose in the script section I get the error above ie “cannot connect to the docker daemon”. This occurs when using tcp://localhost:2375
  2. when using “tcp://docker:2375” it fails in docker login after literally trying to resolve this in DNS and the squid proxy servicing internet connectivity for my environment returns a huge long html error message…

Anyone able to assist here? Everything I am doing is what is supposed to work according to forums and even gitlab’s own documentation yet not working in this instance.

I just tried a new approach based on another example I found.

I set the runner up with:

[runners.docker]
tls_verify = true
image = “alpine:latest”
privileged = true
disable_entrypoint_overwrite = false
oom_kill_disable = false
disable_cache = false
volumes = [“/cache”, “/certs/client” ]

and cut the .gitlab-ci.yml down to just:

merge review:
stage: terraform_plan
image: docker:latest

services:
    - name: docker:dind

However, now I get a new error.

The failing command is: docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY

CI pipeline output:

  • docker login -u gitlab-ci-token -p [MASKED] :5002

WARNING! Using --password via the CLI is insecure. Use --password-stdin.

error during connect: Post “https://docker:2376/v1.24/auth”: Forbidden

I tried adding “docker” to the no_proxy var, result:

  • docker login -u gitlab-ci-token -p [MASKED] :5002

WARNING! Using --password via the CLI is insecure. Use --password-stdin.

error during connect: Post “http://docker:2375/v1.24/auth”: dial tcp: lookup docker on 192.168.0.1:53: no such host

So it clearly tries to lookup “docker” in DNS and of course this fails. I have seen absolutely no mention of anything DNS related in any of the documentation or forum posts and I am completely out of ideas now

I’ve managed to move forwards with this and now get an error regarding the Certificate for the registry being from an unknown issuer.

However, the CA certificate has been installed on the docker image:

In my config.toml file I have:
pre_build_script = “”"
export HTTP_PROXY=http://192.168.0.1:3128/
export HTTPS_PROXY=http://192.168.0.1:3128/
export NO_PROXY=.anfieldroad.int,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16,docker
apk update >/dev/null
apk add ca-certificates > /dev/null
rm -rf /var/cache/apk/*
wget http://ca.anfieldroad.int/certs/anfieldroad-ca-chain-bundle.cert.pem -O /usr/local/share/ca-certificates/anfieldroad-ca-chain-bundle.cert.pem
update-ca-certificates --fresh > /dev/null
“”"

and in my gitlab-ci.yml:
before_script:
- |
set -xv
http_proxy=$http_proxy
https_proxy=$https_proxy
no_proxy=${no_proxy},docker
apk add --no-cache ca-certificates
wget http:///certs/ca-cert.pem -O /usr/local/share/ca-certificates/ca-cert.pem && update-ca-certificates

I believe having looked at the output of the CI job that the two above are working on the same docker container so I can remove one but regardless I still get an error. I believe this is a separate issue to the docker in docker issue so I will create a new topic.