Building Docker images with Dockered Runner and DinD

Let’s start with what I want to achieve: I’d like my pipeline to build Docker images from files in my repo.
I’m using the gitlab/gitlab-runner image on the same host as my gitlab container.
I managed to register the runner and the pipeline starts working but here’s where it all fails.

When I start my runner without -v /var/run/docker.sock:/var/run/docker.sock I get the following error message when invoking the pipeline:

Using Docker executor with image docker:dind …
ERROR: Preparation failed: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

When bind mounting it I get this one:

Cloning into ‘/builds/…’…
fatal: unable to access 'http://gitlab-ci-token:xxxxxxxxxxxxxxxxxxxx@…: SSL certificate problem: unable to get local issuer certificate

Clearly some container is missing the certificate of my gitlab instance. But which? I haven’t grasped the concept of DinD yet. Which container performs which task?

What does the image specified by the image tag in the .gitlab-ci.yml do?
What does the service (docker:dind) specified by the image tag in the .gitlab-ci.yml do?
What does the image specified in the config.toml do?

My .gitlab-ci-yml:

image: docker:stable

variables:
  DOCKER_HOST: tcp://docker:2375/
  DOCKER_DRIVER: overlay2

services:
  - docker:dind

before_script:
  - docker info
# See https://docs.gitlab.com/ce/ci/yaml/README.html
# start with build stage
stages:
  - build
  #- test
  #- deploy

build: # a job with the name build
  stage: build
  tags: # The tags here determine which runner is allowed to run this job. Only runners with the same tag are allowed
    - testtag
  script:
    - docker build -t myImage .
    - docker push myRegistry

My config.toml

concurrent = 1
check_interval = 0

[[runners]]
  name = "Testrunner"
  url = "http://gitlabhost"
  token = "XXX"
  executor = "docker"
  [runners.docker]
    tls_verify = false
    image = "docker:git"
    privileged = true
    disable_cache = false
    volumes = ["/cache"]
    shm_size = 0
  [runners.cache]

What I found out so far is that for the git pull command to work with the Shell Executor I have to add the respective root (and intermediate) CAs to the runner image. For a Dockerfile it’d look like this:

COPY ./myCA.crt /usr/local/share/ca-certificates/
RUN update-ca-certificates

My CA file had to have the extension .crt, not .pem or anything else.

For the Docker Executor, the Runner starts a special continer which performs the cloning.
The main question remains: I don’t know how to add or pass the CAs to that image.
A workaround is to disable the certificate check in the conf.toml but that’s not recommended:

environment = ['GIT_SSL_NO_VERIFY=true']

The image specified under Services is the one running the Docker daemon to which the image specified under image connects when the script calls e.g. docker build.
When using push or pull and the registry uses a custom certificate then this service image must possess the CA similar to the runner image. I built a new image from docker:dind and copied the necessary certificates to

/etc/docker/certs.d/myregistry/ca.crt

I also had to create the directories certs.d all subdirectories.

I’m not sure if the following is related to your issue or not, but I recently discovered that my runners were v9.5.1 , whereas the latest version is v11. I couldn’t clone a repos with submodules with runner v9.5.1 and the error message mislead to a certificate issue, when a runner v11 clones happily.

Have you checked the version of your runner ? (could you share it here as it may help others )

It’s version 11.1:

Version: 11.1.0
Git revision: 081978aa
Git branch:
GO version: go1.8.7
Built: 2018-07-22T07:24:46+00:00
OS/Arch: linux/amd64

I could clone when using the shell executor.