Gitlab runner pipepile error "Cannot connect to the Docker daemon at unix:///var/run/docker.sock"

GitLab CI/CD
,
1

Hi, I am running a job from gitlab through a ci file but the same fails for the following error:

docker build --no-cache --pull --rm --tag 245790544406.dkr.ecr.us-east-1.amazonaws.com/go/promo-codes:ef801485 .

[114](https://gitlab.aws.kiusys.com/go/promo-codes/-/jobs/9293#L114)Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

[115](https://gitlab.aws.kiusys.com/go/promo-codes/-/jobs/9293#L115)make: *** [Makefile:3: build] Error 1

[117](https://gitlab.aws.kiusys.com/go/promo-codes/-/jobs/9293#L117)Cleaning up project directory and file based variables00:01

[119](https://gitlab.aws.kiusys.com/go/promo-codes/-/jobs/9293#L119)ERROR: Job failed: command terminated with exit code 2

Here the configuracion of my gitlab-ci file:


image: docker:20.10.20

services:
  - name: docker:20.10.20-dind
    alias: docker

variables:
  DOCKER_HOST: tcp://docker:2375
  DOCKER_DRIVER: overlay2
  DOCKER_TLS_CERTDIR: ""

stages:
  - test
  - build_dev
  - deploy_dev
  - build_release
  - deploy_stage
  - deploy_prod

.before_script_template:
  before_script:
    - apk add --no-cache make python3 py3-pip
    - pip3 install --upgrade pip
    - pip3 install --no-cache-dir awscli
    - aws ecr get-login-password | docker login --username AWS --password-stdin $DOCKER_REGISTRY

.deploy_before_script_template:
  before_script:
    - apk add --no-cache git jq python3 py3-pip sed
    - pip install yq
    - git config pull.rebase true
    - pip install awscli git-remote-codecommit
    - ASSUMED_ROLE=$(aws sts assume-role --role-arn $CODECOMMIT_ROLE --role-session-name $AWS_ROLE_SESSION_NAME --output json)
    - export AWS_ACCESS_KEY_ID=$(echo $ASSUMED_ROLE | jq -r .Credentials.AccessKeyId)
    - export AWS_SECRET_ACCESS_KEY=$(echo $ASSUMED_ROLE | jq -r .Credentials.SecretAccessKey)
    - export AWS_SESSION_TOKEN=$(echo $ASSUMED_ROLE | jq -r .Credentials.SessionToken)
    - git config --global user.email "gitlab-ci@kiusys.com"
    - git config --global user.name "gitlab-ci"

Test:
  stage: test
  extends: .before_script_template
  script: echo "to do"
  tags: [apps-dev]
  only:
      - develop

build_dev:
  stage: build_dev
  extends: .before_script_template
  script:
    - make build
    - make publish
  tags: [apps-dev]
  only:
      - develop

build_release:
  stage: build_release
  extends: .before_script_template
  script:
    - export RELEASE_TAG=${CI_COMMIT_TAG}
    - make build
    - make publish
  only:
    - /^v\d+\.\d+\.\d+$/

deploy_dev:
  stage: deploy_dev
  dependencies:
    - "build_dev"
  extends: .deploy_before_script_template
  script:
    - export  GITLAB_TOKEN
    - git clone "${CI_SERVER_PROTOCOL}://oauth:${CI_TOKEN}@${CI_SERVER_HOST}/${CI_PROJECT_NAMESPACE}/${DEV_REPO_NAME}.git" --branch=main
    - cd ${DEV_REPO_PATH}
    - yq --arg name "$APP_NAME" --arg image "${DOCKER_REGISTRY}/${ECR_REPO_NAME}:${RELEASE_TAG}" -i -y '(.spec.template.spec.containers[] | select(.name==$name)).image = $image' deployment.yaml
    - git commit -am "Gitlab-ci deploy version ${RELEASE_TAG} from ${CI_COMMIT_REF_NAME}"
    - git push -f origin main
  only:
      - develop

Any helps??

2

pipeline config looks good, so I would look somewhere else. You also haven’t mention which job fails. I see they are using different GitLab Runners. Since you haven’t posted the entire job output it’s hard see what exactly fails, but I suggest to look if the services are actually started or looking into your Runners.

3

Here the entire log for the job.

Running with gitlab-runner 15.11.0 (436955cb)
  on gitlab-runner-5b5ff664f8-4xhbx PeDB51SV, system ID: r_sVCb7HJT9zvz
Preparing the "kubernetes" executor
00:00
Using Kubernetes namespace: gitlab
Using Kubernetes executor with image docker:20.10.20 ...
Using attach strategy to execute scripts...
Preparing environment
00:04
Waiting for pod gitlab/runner-pedb51sv-project-352-concurrent-05df5x to be running, status is Pending
Running on runner-pedb51sv-project-352-concurrent-05df5x via gitlab-runner-5b5ff664f8-4xhbx...
Getting source from Git repository
00:01
Fetching changes with git depth set to 20...
Initialized empty Git repository in /builds/go/promo-codes/.git/
Created fresh repository.
Checking out 8e8eae94 as detached HEAD (ref is develop)...
Skipping Git submodules setup
Executing "step_script" stage of the job script
00:11
$ apk add --no-cache make python3 py3-pip
fetch https://dl-cdn.alpinelinux.org/alpine/v3.16/main/x86_64/APKINDEX.tar.gz
fetch https://dl-cdn.alpinelinux.org/alpine/v3.16/community/x86_64/APKINDEX.tar.gz
(1/24) Installing make (4.3-r0)
(2/24) Installing libbz2 (1.0.8-r1)
(3/24) Installing expat (2.5.0-r0)
(4/24) Installing libffi (3.4.2-r1)
(5/24) Installing gdbm (1.23-r0)
(6/24) Installing xz-libs (5.2.5-r1)
(7/24) Installing libgcc (11.2.1_git20220219-r2)
(8/24) Installing libstdc++ (11.2.1_git20220219-r2)
(9/24) Installing mpdecimal (2.5.1-r1)
(10/24) Installing readline (8.1.2-r0)
(11/24) Installing sqlite-libs (3.40.1-r0)
(12/24) Installing python3 (3.10.13-r0)
(13/24) Installing py3-contextlib2 (21.6.0-r2)
(14/24) Installing py3-tomli (2.0.1-r1)
(15/24) Installing py3-pep517 (0.12.0-r2)
(16/24) Installing py3-six (1.16.0-r1)
(17/24) Installing py3-retrying (1.3.3-r3)
(18/24) Installing py3-appdirs (1.4.4-r3)
(19/24) Installing py3-more-itertools (8.13.0-r0)
(20/24) Installing py3-ordered-set (4.0.2-r3)
(21/24) Installing py3-parsing (2.4.7-r3)
(22/24) Installing py3-packaging (21.3-r0)
(23/24) Installing py3-setuptools (59.4.0-r0)
(24/24) Installing py3-pip (22.1.1-r0)
Executing busybox-1.35.0-r17.trigger
OK: 82 MiB in 46 packages
$ pip3 install --upgrade pip
Requirement already satisfied: pip in /usr/lib/python3.10/site-packages (22.1.1)
Collecting pip
  Downloading pip-23.2.1-py3-none-any.whl (2.1 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.1/2.1 MB 80.7 MB/s eta 0:00:00
Installing collected packages: pip
  Attempting uninstall: pip
    Found existing installation: pip 22.1.1
    Uninstalling pip-22.1.1:
      Successfully uninstalled pip-22.1.1
Successfully installed pip-23.2.1
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
$ pip3 install --no-cache-dir awscli
Collecting awscli
  Obtaining dependency information for awscli from https://files.pythonhosted.org/packages/d3/ef/f3696e2e9dc04077e4d50ce4ad3c237aca5d0068a999cad7da02b129bc43/awscli-1.29.57-py3-none-any.whl.metadata
  Downloading awscli-1.29.57-py3-none-any.whl.metadata (11 kB)
Collecting botocore==1.31.57 (from awscli)
  Obtaining dependency information for botocore==1.31.57 from https://files.pythonhosted.org/packages/1d/48/6c1118ac9168fcb49900a6a1aefb6d70f44da1da689c498293d133a5f5f1/botocore-1.31.57-py3-none-any.whl.metadata
  Downloading botocore-1.31.57-py3-none-any.whl.metadata (6.0 kB)
Collecting docutils<0.17,>=0.10 (from awscli)
  Downloading docutils-0.16-py2.py3-none-any.whl (548 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 548.2/548.2 kB 36.7 MB/s eta 0:00:00
Collecting s3transfer<0.8.0,>=0.7.0 (from awscli)
  Obtaining dependency information for s3transfer<0.8.0,>=0.7.0 from https://files.pythonhosted.org/packages/5a/4b/fec9ce18f8874a96c5061422625ba86c3ee1e6587ccd92ff9f5bf7bd91b2/s3transfer-0.7.0-py3-none-any.whl.metadata
  Downloading s3transfer-0.7.0-py3-none-any.whl.metadata (1.8 kB)
Collecting PyYAML<6.1,>=3.10 (from awscli)
  Obtaining dependency information for PyYAML<6.1,>=3.10 from https://files.pythonhosted.org/packages/07/91/45dfd0ef821a7f41d9d0136ea3608bb5b1653e42fd56a7970532cb5c003f/PyYAML-6.0.1-cp310-cp310-musllinux_1_1_x86_64.whl.metadata
  Downloading PyYAML-6.0.1-cp310-cp310-musllinux_1_1_x86_64.whl.metadata (2.1 kB)
Collecting colorama<0.4.5,>=0.2.5 (from awscli)
  Downloading colorama-0.4.4-py2.py3-none-any.whl (16 kB)
Collecting rsa<4.8,>=3.1.2 (from awscli)
  Downloading rsa-4.7.2-py3-none-any.whl (34 kB)
Collecting jmespath<2.0.0,>=0.7.1 (from botocore==1.31.57->awscli)
  Downloading jmespath-1.0.1-py3-none-any.whl (20 kB)
Collecting python-dateutil<3.0.0,>=2.1 (from botocore==1.31.57->awscli)
  Downloading python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 247.7/247.7 kB 78.4 MB/s eta 0:00:00
Collecting urllib3<1.27,>=1.25.4 (from botocore==1.31.57->awscli)
  Obtaining dependency information for urllib3<1.27,>=1.25.4 from https://files.pythonhosted.org/packages/c5/05/c214b32d21c0b465506f95c4f28ccbcba15022e000b043b72b3df7728471/urllib3-1.26.16-py2.py3-none-any.whl.metadata
  Downloading urllib3-1.26.16-py2.py3-none-any.whl.metadata (48 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 48.4/48.4 kB 193.1 MB/s eta 0:00:00
Collecting pyasn1>=0.1.3 (from rsa<4.8,>=3.1.2->awscli)
  Downloading pyasn1-0.5.0-py2.py3-none-any.whl (83 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 83.9/83.9 kB 207.0 MB/s eta 0:00:00
Requirement already satisfied: six>=1.5 in /usr/lib/python3.10/site-packages (from python-dateutil<3.0.0,>=2.1->botocore==1.31.57->awscli) (1.16.0)
Downloading awscli-1.29.57-py3-none-any.whl (4.3 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 4.3/4.3 MB 118.9 MB/s eta 0:00:00
Downloading botocore-1.31.57-py3-none-any.whl (11.2 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 11.2/11.2 MB 201.6 MB/s eta 0:00:00
Downloading PyYAML-6.0.1-cp310-cp310-musllinux_1_1_x86_64.whl (707 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 707.5/707.5 kB 253.7 MB/s eta 0:00:00
Downloading s3transfer-0.7.0-py3-none-any.whl (79 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 79.8/79.8 kB 229.6 MB/s eta 0:00:00
Downloading urllib3-1.26.16-py2.py3-none-any.whl (143 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 143.1/143.1 kB 225.0 MB/s eta 0:00:00
Installing collected packages: urllib3, PyYAML, python-dateutil, pyasn1, jmespath, docutils, colorama, rsa, botocore, s3transfer, awscli
Successfully installed PyYAML-6.0.1 awscli-1.29.57 botocore-1.31.57 colorama-0.4.4 docutils-0.16 jmespath-1.0.1 pyasn1-0.5.0 python-dateutil-2.8.2 rsa-4.7.2 s3transfer-0.7.0 urllib3-1.26.16
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
$ aws ecr get-login-password | docker login --username AWS --password-stdin $DOCKER_REGISTRY
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store
Login Succeeded
$ docker info
Client:
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc., v0.9.1)
  compose: Docker Compose (Docker Inc., v2.12.2)
Server:
ERROR: Cannot connect to the Docker daemon at tcp://docker:2375. Is the docker daemon running?
errors pretty printing info
Cleaning up project directory and file based variables
00:01
ERROR: Job failed: command terminated with exit code 1
4

So the error message is different this time :slight_smile: and my guess is that the dind cannot start.
In order to run DinD the GitLab Runner must have

[runners.kubernetes]
  privileged = true

Also your Kubernetes cluster must allow privileged pods. Privileged pods are often forbidden, because it’s a security hole.

For Kubernetes environments I suggest to use something else. Docker was not designed to build images in k8s environments. There is kaniko or jib or paketo and other tools which are better.

5

The installation I am doing through helm here my configuration:

gitlabUrl: https://gitlab.xxxx.xxxx.com/
runnerRegistrationToken: “xxxxxx”

rbac:
create: false
serviceAccountName: runner-aws-access

runners:
privileged: true
runUntagged: true
serviceAccountName: runner-aws-access
tags: apps-dev

the error persists any idea how to fix it?

6

Look into job Pod logs in each container to narrow down the root cause