GitLab Runner /// failed to dial gRPC: cannot connect to the Docker daemon. Is 'docker daemon' running on this host?: dial tcp 127.0.0.1:2375: connect: connection refused

Infrastructure as Code & Cloud Native
, , ,
1

Hello World!

I’m trying to get CI/CD going in my GitLab w/ GKE (using default .gitlab-ci.yml (Docker)), yet keep getting following error:

$ docker build --pull -t "$CI_REGISTRY/$CI_REGISTRY_IMAGE" .
time="2019-07-29T03:46:57Z" level=error msg="failed to dial gRPC: cannot connect to the Docker daemon. Is 'docker daemon' running on this host?: dial tcp 127.0.0.1:2375: connect: connection refused"
error during connect: Post http://localhost:2375/v1.40/build?buildargs=%7B%7D&cachefrom=%5B%5D&cgroupparent=&cpuperiod=0&cpuquota=0&cpusetcpus=&cpusetmems=&cpushares=0&dockerfile=Dockerfile&labels=%7B%7D&memory=0&memswap=0&networkmode=default&pull=1&rm=1&session=qaez47rcueaw8it63d4ugsvm3&shmsize=0&t=registry.regemini.com%2Fcommon&target=&ulimits=null&version=1: context canceled

per ### Using docker:dind, I have following

DOCKER_HOST=tcp://localhost:2375

Please advise.

2

Hello, I Have noticed the same issue in Digital Ocean. Its first occurrence was four days ago.

In gitlab runner output I can see:

$ docker build --pull -t "$CI_REGISTRY_IMAGE" .
time="2019-07-29T13:30:15Z" level=error msg="failed to dial gRPC: cannot connect to the Docker daemon. Is 'docker daemon' running on this host?: dial tcp [::1]:2375: connect: connection refused"error during connect: Post http://localhost:2375/v1.40/build?buildargs=%7B%7D&cachefrom=%5B%5D&cgroupparent=&cpuperiod=0&cpuquota=0&cpusetcpus=&cpusetmems=&cpushares=0&dockerfile=Dockerfile&labels=%7B%7D&memory=0&memswap=0&networkmode=default&pull=1&rm=1&session=td7ygu9erizh587l8m3wpejdd&shmsize=0&t=oriongit.foxservice.cz%3A5001%2Forion%2Fservices%2Fapi-proxy&target=&ulimits=null&version=1: context canceled

From gitlab runner logs I can see following:

Registration attempt 1 of 30
Runtime platform                                    arch=amd64 os=linux pid=13 revision=6946bae7 version=12.0.0
WARNING: Running in user-mode.                     
WARNING: The user-mode requires you to manually start builds processing: 
WARNING: $ gitlab-runner run                       
WARNING: Use sudo for system-mode:                 
WARNING: $ sudo gitlab-runner...                   
                                                   
Registering runner... succeeded                     runner=fs-V63t5
Runner registered successfully. Feel free to start it, but if it's running already the config should be automatically reloaded! 
Runtime platform                                    arch=amd64 os=linux pid=20 revision=6946bae7 version=12.0.0
Starting multi-runner from /home/gitlab-runner/.gitlab-runner/config.toml ...  builds=0
WARNING: Running in user-mode.                     
WARNING: Use sudo for system-mode:                 
WARNING: $ sudo gitlab-runner...                   
                                                   
Configuration loaded                                builds=0
Metrics server listening                            address=[::]:9252 builds=0
[session_server].listen_address not defined, session endpoints disabled  builds=0
Checking for jobs... received                       job=878 repo_url=https://redacted/app.git runner=desbPkqf
Job succeeded                                       duration=27.155391888s job=878 project=16 runner=desbPkqf
WARNING: Appending trace to coordinator... aborted  code=403 job=878 job-log= job-status=success runner=desbPkqf sent-log=4548-4628 status=403 Forbidden
WARNING: Submitting job to coordinator... aborted   code=403 job=878 job-status=success runner=desbPkqf
Checking for jobs... received                       job=879 repo_url=https://redacted/app.git runner=desbPkqf
WARNING: Job failed: command terminated with exit code 1  duration=9.467807023s job=879 project=16 runner=desbPkqf
ERROR: Failed to process runner                     builds=0 error=command terminated with exit code 1 executor=kubernetes runner=desbPkqf
Checking for jobs... received                       job=880 repo_url=https://redacted/app.git runner=desbPkqf
WARNING: Job failed: command terminated with exit code 1  duration=9.974310983s job=880 project=32 runner=desbPkqf
ERROR: Failed to process runner                     builds=0 error=command terminated with exit code 1 executor=kubernetes runner=desbPkqf

Anyone any ideas?

Many thanks

3

Probably found a solution:

It worked for me.

Just one note to myself and maybe even for @a1exus,
do not use unversioned images, or images with :latest tag in your pipelines :wink:
e.g.

services:
- docker:dind

or

create docker image latest:
  stage: build
  image: docker:latest
  script:
    ...
5

We had the issue in an AWS EKS Cluster.
After some try and error i figured out that it is an timing issue.
The docker tcp socket is not available yet when the docker command runs.

Found an workaround as described here: