Docker:dind stops working after 12.1.0 update

amatveev · August 2, 2019, 7:32am

Hi,

My gitlab CI pipeline started to fail right after the 12.1.0 release.

I’m using shared-runner and have following gitlab-ci.yaml config:

services:
  - docker:dind

variables:
  DOCKER_DIRVER: "overlay2"
  DOCKER_TLS_CERTDIR: ""
  DOCKER_HOST: tcp://docker:2375

stages:
  - build
  - publish
  - deploy

# Job: Build
# Builds project with gradle

build:
  image: prographerj/centos7-java8:latest
  stage: build
  script:
    - ./gradlew clean build
  tags:
    - docker
  artifacts:
    paths:
      - build/libs/
    expire_in: 1 hour
...

And now i’m getting following error in logs.

Running with gitlab-runner 12.1.0 (de7731dd)
  on docker-auto-scale 72989761
Using Docker executor with image prographerj/centos7-java8:latest ...
Starting service docker:dind ...
Pulling docker image docker:dind ...
Using docker image sha256:381e1d0bc6ef1b9619690dbc0de477588c4a1f14fc781e31b0123ac6f421c25c for docker:dind ...
Waiting for services to be up and running...

*** WARNING: Service runner-72989761-project-12494241-concurrent-0-docker-0 probably didn't start properly.

Health check error:
ContainerStart: Error response from daemon: Cannot link to a non running container: /runner-72989761-project-12494241-concurrent-0-docker-0 AS /runner-72989761-project-12494241-concurrent-0-docker-0-wait-for-service/service (executor_docker.go:1257:0s)

Service container logs:
2019-08-02T06:45:42.560150672Z time="2019-08-02T06:45:42.548131573Z" level=info msg="Starting up"
2019-08-02T06:45:42.560224952Z time="2019-08-02T06:45:42.551847950Z" level=warning msg="[!] DON'T BIND ON ANY IP ADDRESS WITHOUT setting --tlsverify IF YOU DON'T KNOW WHAT YOU'RE DOING [!]"
2019-08-02T06:45:42.567628475Z failed to load listeners: listen tcp: lookup docker on 169.254.169.254:53: no such host

*********
...

Before that (on 12.1.0-rc) everything was fine:

Running with gitlab-runner 12.1.0-rc1 (6da35412)
  on docker-auto-scale ed2dce3a
Using Docker executor with image prographerj/centos7-java8:latest ...
Starting service docker:dind ...
Pulling docker image docker:dind ...
Using docker image sha256:25a1e57c774167d28c44d88fa296f3e1122c6d79e99b98653c899b170393bbd6 for docker:dind ...
Waiting for services to be up and running...
Pulling docker image prographerj/centos7-java8:latest ...
Using docker image sha256:eff847092a3a8fb78f4e2ff93ca739dcbfcda99ff30c38407ae775b17b0013ad for prographerj/centos7-java8:latest ...
Running on runner-ed2dce3a-project-12494241-concurrent-0 via runner-ed2dce3a-srm-1562917026-c98bf791...
...

And everything was fine for like 1 year before.

Maybe this configuration is not relevant anymore?
Thanks in advance.

wouter.sioen · August 2, 2019, 7:53am

I’m having the exact same problem here, since this morning.

timspin · August 2, 2019, 8:17am

I’m also experiencing troubles with docker-dind

mickael · August 2, 2019, 8:19am

Hello, you can fix it temporary by using this:

services:
    - name: docker:dind
      entrypoint: ["env", "-u", "DOCKER_HOST"]
      command: ["dockerd-entrypoint.sh"]
  variables:
    DOCKER_HOST: tcp://docker:2375/
    DOCKER_DRIVER: overlay2
    # See https://github.com/docker-library/docker/pull/166
    DOCKER_TLS_CERTDIR: ""

https://gitlab.com/gitlab-org/gitlab-runner/issues/4566#note_199261985

RaphLH · August 2, 2019, 12:52pm

Saved me some time, thank you !

webmutation · August 2, 2019, 2:27pm

Same exact issue. Problem is with new Docker image, not with Gitlab.

That solved my issue. Thank you!

phpoenx · August 3, 2019, 12:28pm

Using Docker executor with image docker:stable ...
Starting service docker:stable-dind ...
Pulling docker image docker:stable-dind ...
Using docker image sha256:9265d22101aac9ee3b1f8301b352fda34e08904fb4d6ffa3e6e23cddba4f2001 for docker:stable-dind ...
Waiting for services to be up and running...

*** WARNING: Service runner-72989761-project-11865010-concurrent-0-docker-0 probably didn't start properly.

Health check error:
service "runner-72989761-project-11865010-concurrent-0-docker-0-wait-for-service" timeout

Health check container logs:


Service container logs:
2019-08-02T18:38:09.310332440Z time="2019-08-02T18:38:09.310170340Z" level=info msg="Starting up"
2019-08-02T18:38:09.311662084Z time="2019-08-02T18:38:09.311573149Z" level=warning msg="could not change group /var/run/docker.sock to docker: group docker not found"
2019-08-02T18:38:09.311960167Z time="2019-08-02T18:38:09.311904258Z" level=warning msg="[!] DON'T BIND ON ANY IP ADDRESS WITHOUT setting --tlsverify IF YOU DON'T KNOW WHAT YOU'RE DOING [!]"
2019-08-02T18:38:09.314296672Z time="2019-08-02T18:38:09.314227798Z" level=info msg="libcontainerd: started new containerd process" pid=19
2019-08-02T18:38:09.314409960Z time="2019-08-02T18:38:09.314368120Z" level=info msg="parsed scheme: \"unix\"" module=grpc
2019-08-02T18:38:09.314454490Z time="2019-08-02T18:38:09.314429224Z" level=info msg="scheme \"unix\" not registered, fallback to default scheme" module=grpc
2019-08-02T18:38:09.314537037Z time="2019-08-02T18:38:09.314498282Z" level=info msg="ccResolverWrapper: sending update to cc: {[{unix:///var/run/docker/containerd/containerd.sock 0  <nil>}] }" module=grpc
2019-08-02T18:38:09.314581325Z time="2019-08-02T18:38:09.314556111Z" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
2019-08-02T18:38:09.314738318Z time="2019-08-02T18:38:09.314670288Z" level=info msg="pickfirstBalancer: HandleSubConnStateChange: 0xc0001370f0, CONNECTING" module=grpc
2019-08-02T18:38:09.333748246Z time="2019-08-02T18:38:09.333652309Z" level=info msg="starting containerd" revision=894b81a4b802e4eb2a91d1ce216b8817763c29fb version=v1.2.6 
2019-08-02T18:38:09.334146937Z time="2019-08-02T18:38:09.334094111Z" level=info msg="loading plugin "io.containerd.content.v1.content"..." type=io.containerd.content.v1 
2019-08-02T18:38:09.334417745Z time="2019-08-02T18:38:09.334354432Z" level=info msg="loading plugin "io.containerd.snapshotter.v1.btrfs"..." type=io.containerd.snapshotter.v1 
2019-08-02T18:38:09.334780104Z time="2019-08-02T18:38:09.334718198Z" level=warning msg="failed to load plugin io.containerd.snapshotter.v1.btrfs" error="path /var/lib/docker/containerd/daemon/io.containerd.snapshotter.v1.btrfs must be a btrfs filesystem to be used with the btrfs snapshotter" 
2019-08-02T18:38:09.334856126Z time="2019-08-02T18:38:09.334810203Z" level=info msg="loading plugin "io.containerd.snapshotter.v1.aufs"..." type=io.containerd.snapshotter.v1 
2019-08-02T18:38:09.340527164Z time="2019-08-02T18:38:09.340452455Z" level=warning msg="failed to load plugin io.containerd.snapshotter.v1.aufs" error="modprobe aufs failed: "ip: can't find device 'aufs'\nmodprobe: can't change directory to '/lib/modules': No such file or directory\n": exit status 1" 
2019-08-02T18:38:09.340609750Z time="2019-08-02T18:38:09.340553984Z" level=info msg="loading plugin "io.containerd.snapshotter.v1.native"..." type=io.containerd.snapshotter.v1 
2019-08-02T18:38:09.340763847Z time="2019-08-02T18:38:09.340715084Z" level=info msg="loading plugin "io.containerd.snapshotter.v1.overlayfs"..." type=io.containerd.snapshotter.v1 
2019-08-02T18:38:09.340988672Z time="2019-08-02T18:38:09.340941538Z" level=info msg="loading plugin "io.containerd.snapshotter.v1.zfs"..." type=io.containerd.snapshotter.v1 
2019-08-02T18:38:09.341310211Z time="2019-08-02T18:38:09.341255227Z" level=warning msg="failed to load plugin io.containerd.snapshotter.v1.zfs" error="path /var/lib/docker/containerd/daemon/io.containerd.snapshotter.v1.zfs must be a zfs filesystem to be used with the zfs snapshotter" 
2019-08-02T18:38:09.341396786Z time="2019-08-02T18:38:09.341345647Z" level=info msg="loading plugin "io.containerd.metadata.v1.bolt"..." type=io.containerd.metadata.v1 
2019-08-02T18:38:09.341515466Z time="2019-08-02T18:38:09.341454180Z" level=warning msg="could not use snapshotter btrfs in metadata plugin" error="path /var/lib/docker/containerd/daemon/io.containerd.snapshotter.v1.btrfs must be a btrfs filesystem to be used with the btrfs snapshotter" 
2019-08-02T18:38:09.341597247Z time="2019-08-02T18:38:09.341531587Z" level=warning msg="could not use snapshotter aufs in metadata plugin" error="modprobe aufs failed: "ip: can't find device 'aufs'\nmodprobe: can't change directory to '/lib/modules': No such file or directory\n": exit status 1" 
2019-08-02T18:38:09.341640286Z time="2019-08-02T18:38:09.341612787Z" level=warning msg="could not use snapshotter zfs in metadata plugin" error="path /var/lib/docker/containerd/daemon/io.containerd.snapshotter.v1.zfs must be a zfs filesystem to be used with the zfs snapshotter" 
2019-08-02T18:38:09.357133710Z time="2019-08-02T18:38:09.353867372Z" level=info msg="loading plugin "io.containerd.differ.v1.walking"..." type=io.containerd.differ.v1 
2019-08-02T18:38:09.357145787Z time="2019-08-02T18:38:09.353890320Z" level=info msg="loading plugin "io.containerd.gc.v1.scheduler"..." type=io.containerd.gc.v1 
2019-08-02T18:38:09.357163026Z time="2019-08-02T18:38:09.353919956Z" level=info msg="loading plugin "io.containerd.service.v1.containers-service"..." type=io.containerd.service.v1 
2019-08-02T18:38:09.357166089Z time="2019-08-02T18:38:09.353931587Z" level=info msg="loading plugin "io.containerd.service.v1.content-service"..." type=io.containerd.service.v1 
2019-08-02T18:38:09.357182071Z time="2019-08-02T18:38:09.353942037Z" level=info msg="loading plugin "io.containerd.service.v1.diff-service"..." type=io.containerd.service.v1 
2019-08-02T18:38:09.357185440Z time="2019-08-02T18:38:09.353952648Z" level=info msg="loading plugin "io.containerd.service.v1.images-service"..." type=io.containerd.service.v1 
2019-08-02T18:38:09.357188230Z time="2019-08-02T18:38:09.353963408Z" level=info msg="loading plugin "io.containerd.service.v1.leases-service"..." type=io.containerd.service.v1 
2019-08-02T18:38:09.357191198Z time="2019-08-02T18:38:09.353973714Z" level=info msg="loading plugin "io.containerd.service.v1.namespaces-service"..." type=io.containerd.service.v1 
2019-08-02T18:38:09.357194074Z time="2019-08-02T18:38:09.353982905Z" level=info msg="loading plugin "io.containerd.service.v1.snapshots-service"..." type=io.containerd.service.v1 
2019-08-02T18:38:09.357196950Z time="2019-08-02T18:38:09.354006069Z" level=info msg="loading plugin "io.containerd.runtime.v1.linux"..." type=io.containerd.runtime.v1 
2019-08-02T18:38:09.357199904Z time="2019-08-02T18:38:09.354143998Z" level=info msg="loading plugin "io.containerd.runtime.v2.task"..." type=io.containerd.runtime.v2 
2019-08-02T18:38:09.357202791Z time="2019-08-02T18:38:09.354229036Z" level=info msg="loading plugin "io.containerd.monitor.v1.cgroups"..." type=io.containerd.monitor.v1 
2019-08-02T18:38:09.357211069Z time="2019-08-02T18:38:09.354538374Z" level=info msg="loading plugin "io.containerd.service.v1.tasks-service"..." type=io.containerd.service.v1 
2019-08-02T18:38:09.357214249Z time="2019-08-02T18:38:09.354563852Z" level=info msg="loading plugin "io.containerd.internal.v1.restart"..." type=io.containerd.internal.v1 
2019-08-02T18:38:09.357217537Z time="2019-08-02T18:38:09.354600938Z" level=info msg="loading plugin "io.containerd.grpc.v1.containers"..." type=io.containerd.grpc.v1 
2019-08-02T18:38:09.357220276Z time="2019-08-02T18:38:09.354611546Z" level=info msg="loading plugin "io.containerd.grpc.v1.content"..." type=io.containerd.grpc.v1 
2019-08-02T18:38:09.357222849Z time="2019-08-02T18:38:09.354620830Z" level=info msg="loading plugin "io.containerd.grpc.v1.diff"..." type=io.containerd.grpc.v1 
2019-08-02T18:38:09.357225601Z time="2019-08-02T18:38:09.354630259Z" level=info msg="loading plugin "io.containerd.grpc.v1.events"..." type=io.containerd.grpc.v1 
2019-08-02T18:38:09.357228197Z time="2019-08-02T18:38:09.354639574Z" level=info msg="loading plugin "io.containerd.grpc.v1.healthcheck"..." type=io.containerd.grpc.v1 
2019-08-02T18:38:09.357230897Z time="2019-08-02T18:38:09.354649038Z" level=info msg="loading plugin "io.containerd.grpc.v1.images"..." type=io.containerd.grpc.v1 
2019-08-02T18:38:09.357233446Z time="2019-08-02T18:38:09.354656941Z" level=info msg="loading plugin "io.containerd.grpc.v1.leases"..." type=io.containerd.grpc.v1 
2019-08-02T18:38:09.357236129Z time="2019-08-02T18:38:09.354665809Z" level=info msg="loading plugin "io.containerd.grpc.v1.namespaces"..." type=io.containerd.grpc.v1 
2019-08-02T18:38:09.357238719Z time="2019-08-02T18:38:09.354675539Z" level=info msg="loading plugin "io.containerd.internal.v1.opt"..." type=io.containerd.internal.v1 
2019-08-02T18:38:09.357241323Z time="2019-08-02T18:38:09.354910534Z" level=info msg="loading plugin "io.containerd.grpc.v1.snapshots"..." type=io.containerd.grpc.v1 
2019-08-02T18:38:09.357243927Z time="2019-08-02T18:38:09.354927366Z" level=info msg="loading plugin "io.containerd.grpc.v1.tasks"..." type=io.containerd.grpc.v1 
2019-08-02T18:38:09.357257434Z time="2019-08-02T18:38:09.354937827Z" level=info msg="loading plugin "io.containerd.grpc.v1.version"..." type=io.containerd.grpc.v1 
2019-08-02T18:38:09.357260646Z time="2019-08-02T18:38:09.354947436Z" level=info msg="loading plugin "io.containerd.grpc.v1.introspection"..." type=io.containerd.grpc.v1 
2019-08-02T18:38:09.357263465Z time="2019-08-02T18:38:09.355143147Z" level=info msg=serving... address="/var/run/docker/containerd/containerd-debug.sock" 
2019-08-02T18:38:09.357266184Z time="2019-08-02T18:38:09.355202005Z" level=info msg=serving... address="/var/run/docker/containerd/containerd.sock" 
2019-08-02T18:38:09.357268761Z time="2019-08-02T18:38:09.355210869Z" level=info msg="containerd successfully booted in 0.022105s" 
2019-08-02T18:38:09.374070286Z time="2019-08-02T18:38:09.371656128Z" level=info msg="pickfirstBalancer: HandleSubConnStateChange: 0xc0001370f0, READY" module=grpc
2019-08-02T18:38:09.374084699Z time="2019-08-02T18:38:09.373102323Z" level=info msg="Setting the storage driver from the $DOCKER_DRIVER environment variable (overlay2)"
2019-08-02T18:38:09.374095704Z time="2019-08-02T18:38:09.373307888Z" level=info msg="parsed scheme: \"unix\"" module=grpc
2019-08-02T18:38:09.374098990Z time="2019-08-02T18:38:09.373320233Z" level=info msg="scheme \"unix\" not registered, fallback to default scheme" module=grpc
2019-08-02T18:38:09.374102084Z time="2019-08-02T18:38:09.373337708Z" level=info msg="ccResolverWrapper: sending update to cc: {[{unix:///var/run/docker/containerd/containerd.sock 0  <nil>}] }" module=grpc
2019-08-02T18:38:09.374105457Z time="2019-08-02T18:38:09.373345923Z" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
2019-08-02T18:38:09.374108401Z time="2019-08-02T18:38:09.373402661Z" level=info msg="pickfirstBalancer: HandleSubConnStateChange: 0xc00060f1e0, CONNECTING" module=grpc
2019-08-02T18:38:09.374111630Z time="2019-08-02T18:38:09.373748580Z" level=info msg="pickfirstBalancer: HandleSubConnStateChange: 0xc00060f1e0, READY" module=grpc
2019-08-02T18:38:09.404764077Z time="2019-08-02T18:38:09.389608378Z" level=info msg="parsed scheme: \"unix\"" module=grpc
2019-08-02T18:38:09.404780444Z time="2019-08-02T18:38:09.389626276Z" level=info msg="scheme \"unix\" not registered, fallback to default scheme" module=grpc
2019-08-02T18:38:09.404784456Z time="2019-08-02T18:38:09.389644258Z" level=info msg="ccResolverWrapper: sending update to cc: {[{unix:///var/run/docker/containerd/containerd.sock 0  <nil>}] }" module=grpc
2019-08-02T18:38:09.404788144Z time="2019-08-02T18:38:09.389653129Z" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
2019-08-02T18:38:09.404791374Z time="2019-08-02T18:38:09.389801082Z" level=info msg="pickfirstBalancer: HandleSubConnStateChange: 0xc00060f7f0, CONNECTING" module=grpc
2019-08-02T18:38:09.404794456Z time="2019-08-02T18:38:09.394710011Z" level=info msg="pickfirstBalancer: HandleSubConnStateChange: 0xc00060f7f0, READY" module=grpc
2019-08-02T18:38:09.508550918Z time="2019-08-02T18:38:09.491036761Z" level=info msg="Loading containers: start."
2019-08-02T18:38:09.508572087Z time="2019-08-02T18:38:09.507275472Z" level=warning msg="Running modprobe bridge br_netfilter failed with message: ip: can't find device 'bridge'\nbridge                167936  1 br_netfilter\nstp                    16384  1 bridge\nllc                    16384  2 bridge,stp\nip: can't find device 'br_netfilter'\nbr_netfilter           24576  0 \nbridge                167936  1 br_netfilter\nmodprobe: can't change directory to '/lib/modules': No such file or directory\n, error: exit status 1"
2019-08-02T18:38:09.512382580Z time="2019-08-02T18:38:09.512284214Z" level=warning msg="Running modprobe nf_nat failed with message: `ip: can't find device 'nf_nat'\nnf_nat_ipv4            16384  2 ipt_MASQUERADE,iptable_nat\nnf_nat                 32768  1 nf_nat_ipv4\nnf_conntrack          135168  5 ipt_MASQUERADE,nf_conntrack_netlink,nf_nat_ipv4,nf_nat,xt_conntrack\nlibcrc32c              16384  2 nf_nat,nf_conntrack\nmodprobe: can't change directory to '/lib/modules': No such file or directory`, error: exit status 1"
2019-08-02T18:38:09.529149655Z time="2019-08-02T18:38:09.524373190Z" level=warning msg="Running modprobe xt_conntrack failed with message: `ip: can't find device 'xt_conntrack'\nxt_conntrack           16384  2 \nnf_conntrack          135168  5 ipt_MASQUERADE,nf_conntrack_netlink,nf_nat_ipv4,nf_nat,xt_conntrack\nmodprobe: can't change directory to '/lib/modules': No such file or directory`, error: exit status 1"
2019-08-02T18:38:09.615502291Z time="2019-08-02T18:38:09.615346880Z" level=info msg="Default bridge (docker0) is assigned with an IP address 172.18.0.0/16. Daemon option --bip can be used to set a preferred IP address"
2019-08-02T18:38:09.661131259Z time="2019-08-02T18:38:09.660972478Z" level=info msg="Loading containers: done."
2019-08-02T18:38:09.683191783Z time="2019-08-02T18:38:09.682828424Z" level=info msg="Docker daemon" commit=74b1e89e8a graphdriver(s)=overlay2 version=19.03.1
2019-08-02T18:38:09.683207508Z time="2019-08-02T18:38:09.682976590Z" level=info msg="Daemon has completed initialization"
2019-08-02T18:38:09.725916931Z time="2019-08-02T18:38:09.725586149Z" level=info msg="API listen on [::]:2375"
2019-08-02T18:38:09.725939300Z time="2019-08-02T18:38:09.725669222Z" level=info msg="API listen on /var/run/docker.sock"

*********

Pulling docker image docker:stable ...

I found error in logs too, but dind works normally

4k1l · August 5, 2019, 12:41pm

@phpoenx I am getting the same errors as well

prawana.perera · September 16, 2019, 4:25pm

I’m seeing the same issue
Service container logs:
2019-08-02T07:59:10.251227003Z time=“2019-08-02T07:59:10.218919361Z” level=info msg=“Starting up”
2019-08-02T07:59:10.252744270Z time=“2019-08-02T07:59:10.219453729Z” level=warning msg="[!] DON’T BIND ON ANY IP ADDRESS WITHOUT setting --tlsverify IF YOU DON’T KNOW WHAT YOU’RE DOING [!]"
2019-08-02T07:59:10.252749916Z failed to load listeners: listen tcp: lookup docker on 169.254.169.254:53: no such host

pheanex · August 15, 2020, 4:28am

Still happening in 13.2.2
Are there any fixes in sight?

lucile_gabrielle · August 27, 2020, 8:37am

It is working now, don’t forget --docker-privileged and keep docker and docker-dind versions coherent:

job:login:
  image: docker:latest
  stage: login
  tags:
    - docker-latest-dind
  services:
    - docker:dind
  before_script:
    - docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" $CI_REGISTRY
  script: ....

Working with the – docker-privileged of same version latest

gitlab-runner register -n \
  --non-interactive \
  --url https://YOUR_gitlab.com/ \
  --registration-token YOUR_REGISTRATION_TOKEN \
  --name YOUR_RUNNER_NAME
  --tag-list "docker-latest-dind" \
  --executor docker \
  --docker-image "docker:latest" \
  --docker-privileged

======== you can precise the version it is working with 19.03.12 or 18.09.7 ====

job:login19:
  image: docker:19.03.12
  stage: login
  tags:
    - docker-19.03.12-dind
  services:
    - docker:19.03.12-dind
  before_script:
    - docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" $CI_REGISTRY
  script: ....

Working with the – docker-privileged of same version ex: 19.03.12

gitlab-runner register -n \
  --non-interactive \
  --url https://YOUR_gitlab.com/ \
  --registration-token YOUR_REGISTRATION_TOKEN \
  --name YOUR_RUNNER_NAME
  --tag-list "docker-19.03.12-dind" \
  --executor docker \
  --docker-image "docker:19.03.12" \
  --docker-privileged

Topic		Replies	Views
Dind gitlab-runner service doesn't start up suddenly How to Use GitLab docker	9	18349	September 16, 2019
DinD failures after upgrade GitLab CI/CD ci , runner , docker	2	396	September 4, 2019
Is the docker daemon running? - Did something change with docker dind services? GitLab CI/CD ci , runner , docker , pipelines	1	2900	November 11, 2020
DinD (docker in docker) not working with Centos7.8, docker 19.03.8 and gitlab-runner 12.10.1 GitLab CI/CD	0	1540	May 5, 2020
Unable to build docker images on shared runners GitLab CI/CD ci , runner	5	2510	January 18, 2024

Docker:dind stops working after 12.1.0 update

Related topics