Having gitlab ci error with k8s gitlab runner msg="failed to dial gRPC: cannot connect to the Docker daemon. Is 'docker daemon' running on this host?: dial tcp 127.0.0.1:2375: connect: connection refused"

ali.j.alhajj · July 26, 2022, 11:51am

Replace this template with your information

Having gitlab ci error with k8s gitlab runner

msg=“failed to dial gRPC: cannot connect to the Docker daemon. Is ‘docker daemon’ running on this host?: dial tcp 127.0.0.1:2375: connect: connection refused”
Consider including screenshots, error messages, and/or other helpful visuals
gitlab runner v13.0.0 and am using the gitlab.com no a self managed gitlab

Steps used to install and configure gitlab runner on k8s

$ kubectl create ns gitlab-runner

$ nano role.yaml

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: gitlab-runner
namespace: gitlab-runner
rules:

apiGroups: [“”]
resources: [“*”]
verbs: [“list”, “get”, “watch”, “create”, “delete”, “patch”, “update”]
apiGroups: [“”]
resources: [“pods/exec”]
verbs: [“create”]
apiGroups: [“”]
resources: [“pods/log”]
verbs: [“get”]


$ kubectl create -f role.yaml
$ kubectl create rolebinding --namespace=gitlab-runner gitlab-runner-binding --role=gitlab-runner --serviceaccount=gitlab-runner:default

From Repo settings --> CICD --> Runners

Take the gitlab url and the token and put them in file values.yaml
$ nano values.yaml

gitlabUrl: https://gitlab.com/
runnerRegistrationToken: “”




$ helm repo add gitlab https://charts.gitlab.io

$ helm search repo -l gitlab/gitlab-runner

$ helm install --namespace gitlab-runner gitlab-runner -f values.yaml gitlab/gitlab-runner --version 0.17.0

Now in the CI CD settings of the repo the new runner will be enabled automatically, also edit it and add as tags to it "kubernetes" to make it known. 

Trigger the pipeline and monitor the build and deploy stages.

gitlab ci file:


image: docker:19.03.1

services:
  - docker:19.03.1-dind

variables:
  DOCKER_DRIVER: overlay2
  DOCKER_HOST: tcp://localhost:2375
  DOCKER_TLS_CERTDIR: ''
  GIT_SUBMODULE_STRATEGY: recursive

stages:
  - build
  - deploy

## Templates

.build:
  stage: build
  tags:
    - kubernetes
  script:
    - cd $SOURCE_PATH
    - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
    - docker build -t "$IMAGE" -t "$CI_REGISTRY_IMAGE/$ENV:latest" --build-arg NODE_ENV=${ENV} --network host .
    - docker push "$IMAGE"
    - docker push "$CI_REGISTRY_IMAGE/$ENV:latest"

.deploy:
  stage: deploy
  image: alpine
  tags:
    - kubernetes
  script:
    - cd $SOURCE_PATH
    - apk add --no-cache curl gettext
    - curl -LO https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/linux/amd64/kubectl
    - chmod +x ./kubectl
    - mv ./kubectl /usr/local/bin/kubectl
    - envsubst < $K8S_APP_CONFIG > ./k8s.yml
    - kubectl apply -f k8s.yml

.stage:
  variables:
    ENV: 'stage'
    REPLICAS: 1
    HOST_URL: $ENV.test-mig.greple.ai
  only:
    refs:
      - stage

.prod:
  variables:
    ENV: 'prod'
    REPLICAS: 1
    HOST_URL: test-mig.greple.ai
  only:
    refs:
      - prod

.app:
  variables:
    APP_NAME: 'test-mig'
    K8S_APP_CONFIG: 'ci/k8s-services.yml'
    CI_REGISTRY_IMAGE: 'registry.gitlab.com/greple/migration-test-repo'
    IMAGE: $CI_REGISTRY_IMAGE/$ENV:$CI_PIPELINE_IID
    SOURCE_PATH: '.'
    PORT: 3000
    NAMESPACE: $APP_NAME-$ENV

build-prod:
  extends:
    - .app
    - .prod
    - .build

deploy-prod:
  extends:
    - .app
    - .prod
    - .deploy
  when: manual
  needs:
    - build-prod

build-stage:
  extends:
    - .app
    - .stage
    - .build

deploy-stage:
  extends:
    - .app
    - .stage
    - .deploy
  needs:
    - build-stage

gitlab ci pipeline output with error:

Running with gitlab-runner 13.0.0 (c127439c)
  on gitlab-runner-gitlab-runner-6f546b5ffc-74hdf kMHKtyLY
Preparing the "kubernetes" executor
00:00
Using Kubernetes namespace: gitlab-runner
Using Kubernetes executor with image docker:19.03.1 ...
Preparing environment
00:04
Waiting for pod gitlab-runner/runner-kmhktyly-project-38049572-concurrent-06l2kc to be running, status is Pending
Running on runner-kmhktyly-project-38049572-concurrent-06l2kc via gitlab-runner-gitlab-runner-6f546b5ffc-74hdf...
Getting source from Git repository
00:02
Fetching changes with git depth set to 20...
Initialized empty Git repository in /builds/greple/migration-test-repo/.git/
Created fresh repository.
From https://gitlab.com/greple/migration-test-repo
 * [new ref]         refs/pipelines/595921335 -> refs/pipelines/595921335
 * [new branch]      stage                    -> origin/stage
Checking out 37e5a56c as stage...
Updating/initializing submodules recursively...
Restoring cache
00:00
Downloading artifacts
00:00
Running before_script and script
00:01
$ cd $SOURCE_PATH
$ docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
WARNING! Your password will be stored unencrypted in [MASKED]/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store
Login Succeeded
$ docker build -t "$IMAGE" -t "$CI_REGISTRY_IMAGE/$ENV:latest" --build-arg NODE_ENV=${ENV} --network host .
time="2022-07-25T10:32:48Z" level=error msg="failed to dial gRPC: cannot connect to the Docker daemon. Is 'docker daemon' running on this host?: dial tcp 127.0.0.1:2375: connect: connection refused"
error during connect: Post http://localhost:2375/v1.40/build?buildargs=%7B%22NODE_ENV%22%3A%22stage%22%7D&cachefrom=%5B%5D&cgroupparent=&cpuperiod=0&cpuquota=0&cpusetcpus=&cpusetmems=&cpushares=0&dockerfile=Dockerfile&labels=%7B%7D&memory=0&memswap=0&networkmode=host&rm=1&session=8oy7p15r97862mzbei2n6hq2f&shmsize=0&t=registry.gitlab.com%2Fgreple%2Fmigration-test-repo%2Fstage%3A55&t=registry.gitlab.com%2Fgreple%2Fmigration-test-repo%2Fstage%3Alatest&target=&ulimits=null&version=1: context canceled
Running after_script
00:00
Uploading artifacts for failed job
00:00
ERROR: Job failed: command terminated with exit code 1

Tried to use a lot of different versions of both the dind and docker, and also the gitlab runner installed on k8s. In addition to changing the docker host port and even erasing it.

Thanks in advance

balonik · July 26, 2022, 1:30pm

Hi @ali.j.alhajj

Regarding using global keywords, they are deprecated

I don’t see in your Job log that any service is actually starting. Why not just add the DinD only for .build job? As far as I see you don’t need it anywhere else.

I will also recommend using some k8s-native container building tool like kaniko. See GitLab docs. Instead of spinning up Docker daemon inside k8s.

ali.j.alhajj · July 27, 2022, 7:48am

Hello @balonik , thanks for your reply. I put the dind and the image and services inside the .build block and i got the same log output

ali.j.alhajj · July 27, 2022, 9:59am

@balonik even adding this to the .build block is producing the same error

.build:
  stage: build
  tags:
    - kubernetes
  script:
    - cd $SOURCE_PATH
    - apk update
    - wget https://download.docker.com/linux/static/stable/x86_64/docker-19.03.1.tgz
    - tar xzvf docker-19.03.1.tgz
    - cp docker/* /usr/bin/
    - dockerd &
    - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
    - docker build -t "$IMAGE" -t "$CI_REGISTRY_IMAGE/$ENV:latest" --build-arg NODE_ENV=${ENV} --network host .
    - docker push "$IMAGE"
    - docker push "$CI_REGISTRY_IMAGE/$ENV:latest"

ali.j.alhajj · July 27, 2022, 11:30am

Also adding the image as ubuntu instead of docker for the whole file and using this block it’s giving the same error:

.build:
  stage: build
  tags:
    - kubernetes
  script:
    - cd $SOURCE_PATH
    - echo y | apt update
    #- echo y | apt install wget
    - echo y | apt-get update
    # - wget https://download.docker.com/linux/static/stable/x86_64/docker-19.03.1.tgz
    # - tar xzvf docker-19.03.1.tgz
    # - cp docker/* /usr/bin/
    - echo y | apt install docker.io
   # - service docker start
    - docker run hello-word
    - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
    - docker build -t "$IMAGE" -t "$CI_REGISTRY_IMAGE/$ENV:latest" --build-arg NODE_ENV=${ENV} --network host .
    - docker push "$IMAGE"
    - docker push "$CI_REGISTRY_IMAGE/$ENV:latest"

ali.j.alhajj · July 27, 2022, 12:43pm

Fixed it. The issue was that in the gitlab runner deployment I should define a key/value pair variable KUBERNETES_PRIVILEGED as true.

venerari · January 13, 2023, 3:04pm

This right?
[[runners]]
[runners.kubernetes]
privileged = true

Topic		Replies	Views
GitLab Runner /// failed to dial gRPC: cannot connect to the Docker daemon. Is 'docker daemon' running on this host?: dial tcp 127.0.0.1:2375: connect: connection refused Infrastructure as Code & Cloud Native ci , runner , docker , kubernetes	3	16954	July 2, 2021
Runner in the K8S-cluster: "Is 'docker daemon' running on this host?: dial tcp 127.0.0.1:2375: connect: connection refused" Infrastructure as Code & Cloud Native ci , runner , docker , kubernetes	1	1180	July 19, 2023
Gitlab shared runner can not connect to Docker daemon and leaks secrets GitLab CI/CD ci , runner , docker	7	29455	July 23, 2019
Gitlab kubernetes runner cannot connect to the docker daemon Infrastructure as Code & Cloud Native ci , runner , docker , kubernetes	0	572	December 4, 2019
Socket error when GitLab CI/CD runner , docker , kubernetes	0	28	October 1, 2024

Having gitlab ci error with k8s gitlab runner msg="failed to dial gRPC: cannot connect to the Docker daemon. Is 'docker daemon' running on this host?: dial tcp 127.0.0.1:2375: connect: connection refused"

Replace this template with your information

Related topics