Unable to build docker images on shared runners

GitLab CI/CD
,
1

I am using shared Gitlab runners to build a docker images using dind, see pipeline below. I was using this pipeline for more than two years without any issues.

.buildDocker:
  stage: build-docker
  only:
    - master
  image: docker
  services:
    - docker:dind
  before_script:
    - docker login -u gitlab-ci-token -p $CI_ACCESS_TOKEN registry.gitlab.com
  script:
    - docker pull $CI_REGISTRY_IMAGE:latest || true
    - docker build --cache-from $CI_REGISTRY_IMAGE:latest -t $CI_REGISTRY_IMAGE:${CI_COMMIT_SHA:0:12} -t $CI_REGISTRY_IMAGE:latest -f Dockerfile .
    - docker push $CI_REGISTRY_IMAGE:${CI_COMMIT_SHA:0:12}
    - docker push $CI_REGISTRY_IMAGE:latest

However today the pipeline stopped working (yesterday it worked), error:

$ docker pull $CI_REGISTRY_IMAGE:latest || true
Cannot connect to the Docker daemon at tcp://docker:2375. Is the docker daemon running?
$ docker build --cache-from $CI_REGISTRY_IMAGE:latest -t $CI_REGISTRY_IMAGE:${CI_COMMIT_SHA:0:12} -t $CI_REGISTRY_IMAGE:latest -f Dockerfile .
Cannot connect to the Docker daemon at tcp://docker:2375. Is the docker daemon running?

I have discovered, that gitlab runner version changed from 16.6.0~beta.105.gd2263193 to 16.8.0~beta.67.gb5664e8e

There is also a warning, that did not appear in previous version:

*** WARNING: Service runner-ns46nmmj-project-46375469-concurrent-0-3fdfe577c431c0a2-docker-0 probably didn't start properly.
Health check error:
create service container: "specify container image platform" requires API version 1.41, but the Docker daemon API version is 1.40 (services.go:192:0s)
Service container logs:
2024-01-18T13:46:25.311174781Z cat: can't open '/proc/net/arp_tables_names': No such file or directory
2024-01-18T13:46:25.313622556Z ip: can't find device 'nf_tables'
2024-01-18T13:46:25.314617549Z modprobe: can't change directory to '/lib/modules': No such file or directory
*********

Does anyone have similar issue?
How to fix this?

Tried approaches (that did not work)

  • changed image to order image: docker:20.10.16 and service to docker:20.10.16-dind
  • tried latest tag docker:24 and docker:24-dind
2

Thanks for reporting. Same in Healthcheck failed - requires API version 1.41, but the Docker daemon API version is 1.40 I have asked internally.

1 Like
3

Ok, it seems like the issue is duplicated now. Should I close this one and keep an eye on the other one?

4

No worries. Forum topics will not be closed. I only meant to share additional references so that everyone who reads this topic, also finds the other one. You can subscribe to the other topic for updates, through the action at the bottom, changed to watching.

image

1 Like
5

FYI - You can follow incident issue in 2024-01-18: DIND not working in Shared Runners after update (#17422) · Issues · GitLab.com / GitLab Infrastructure Team / production · GitLab and track its status, and on GitLab System Status

1 Like
6

FYI - operational again. Sorry for the inconvenience, and thanks for reporting.

1 Like