Pipeline job with docker:dind failed on self hosted runner, but success on gitlab.com runner?

GitLab CI/CD
, , ,
1

I try to configure self hosted Gitlab and Gitlab runner, both run on docker based on this and this on Ubuntu server 20.04 LTS with docker.io engine (not the distro package). I test the deployment by mirroring some projects from gitlab.com and to see whether the pipeline run normally or not.

Unfortunately, all pipeline jobs on self hosted runner always failed when the job using docker:dind services, although it works normal on gitlab.com. jobs with other base image, like php:8.0 or postgres:14-alpine ran smoothly.

Here is the gitlab runner config:

concurrent = 1
check_interval = 0

[session_server]
  session_timeout = 1800

[[runners]]
  name = "gitlabrunner"
  url = "http://gitlab-domain-name/"
  token = "redacted"
  executor = "docker"
  [runners.custom_build_dir]
  [runners.cache]
    [runners.cache.s3]
    [runners.cache.gcs]
    [runners.cache.azure]
  [runners.docker]
    tls_verify = false
    image = "ubuntu:latest"
    privileged = true
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = false
    volumes = ["/cache"]
    shm_size = 0

And here is the .gitlab-ci.yml file:

# You can override the included template(s) by including variable overrides
# SAST customization: https://docs.gitlab.com/ee/user/application_security/sast/#customizing-the-sast-settings
# Secret Detection customization: https://docs.gitlab.com/ee/user/application_security/secret_detection/#customizing-settings
# Note that environment variables can be set in several places
# See https://docs.gitlab.com/ee/ci/variables/#cicd-variable-precedence
image: php:8.0

stages:
  - build
  - test
  - deploy
  - review
  - dast
  - staging
  - canary
  - production
  - incremental rollout 10%
  - incremental rollout 25%
  - incremental rollout 50%
  - incremental rollout 100%
  - performance
  - cleanup
  - release

variables:
  POSTGRES_USER: user
  POSTGRES_PASSWORD: pass
  POSTGRES_DB: name
  DOCKER_DRIVER: overlay2
  DOCKER_TLS_CERTDIR: ""

services:
  - name: postgres:14-alpine
    alias: database
  - name: redis:latest
    alias: redis

build_backend:
  image: docker:dind
  stage: build
  variables:
    IMAGE_TAG_BACKEND: $CI_REGISTRY_IMAGE/backend:$CI_COMMIT_REF_SLUG
    DOCKER_TLS_CERTDIR: ''
  services:
    - name: 'docker:20.10.6-dind'
      command: [ '--tls=false', '--host=tcp://0.0.0.0:2375' ]
      alias: docker
  script:
    - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
    - docker build --no-cache --target backend --build-arg STABILITY=-stable -t backend .
    - docker tag backend $IMAGE_TAG_BACKEND
    - docker push $IMAGE_TAG_BACKEND

build_nginx:
  image: docker:dind
  stage: build
  variables:
    IMAGE_TAG: $CI_REGISTRY_IMAGE/nginx:$CI_COMMIT_REF_SLUG
    DOCKER_TLS_CERTDIR: ''
  services:
    - name: 'docker:20.10.6-dind'
      command: ['--tls=false', '--host=tcp://0.0.0.0:2375']
      alias: docker
  script:
    - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
    - docker build --no-cache --target nginx -t nginx .
    - docker tag nginx $IMAGE_TAG
    - docker push $IMAGE_TAG

test:
  image: php:8.0
  variables:
    POSTGRES_USER: user
    POSTGRES_PASSWORD: pass
    POSTGRES_DB: name
    DOCKER_DRIVER: overlay2
    DOCKER_TLS_CERTDIR: ""
  services:
    - name: postgres:14-alpine
      alias: database
    - name: redis:latest
      alias: redis
  before_script:
    - apt-get update -yqq
    - apt-get install git nodejs unzip libcurl4-gnutls-dev libicu-dev libonig-dev libzip-dev
      libmcrypt-dev libvpx-dev libjpeg-dev libpng-dev libxpm-dev zlib1g-dev libfreetype6-dev
      libxml2-dev libexpat1-dev libbz2-dev libgmp3-dev libldap2-dev unixodbc-dev libpq-dev
      libsqlite3-dev libaspell-dev libsnmp-dev libpcre3-dev libtidy-dev -yqq
    - docker-php-ext-install pdo_pgsql zip
    - pecl install xdebug
    - docker-php-ext-enable xdebug
    - curl -sS https://getcomposer.org/installer | php
    - curl -sS https://get.symfony.com/cli/installer | bash
    - export PATH="$HOME/.symfony/bin:$PATH"
    - symfony composer install
    - symfony console doctrine:schema:create
    - symfony console doctrine:fixtures:load --no-interaction
    - symfony server:ca:install
    - symfony serve -d
    - echo "xdebug.mode=develop,coverage,debug" >> /usr/local/etc/php/php.ini
  script:
    - php bin/phpunit --coverage-text --colors=never --log-junit test-report.xml
  cache:
    key: cacheTest
    paths:
      - vendor/
      - node_modules/
  artifacts:
    when: always
    reports:
      junit: test-report.xml

sast:
  stage: test

workflow:
  rules:
    - if: '$BUILDPACK_URL || $AUTO_DEVOPS_EXPLICITLY_ENABLED == "1" || $DOCKERFILE_PATH'

dast:
  variables:
    DAST_FULL_SCAN_ENABLED: "true"
    DAST_WEBSITE: "$DAST_WEBSITE"

deploy_package:
  stage: deploy
  before_script:
    - apt-get update -y && apt dist-upgrade -y && apt install curl -y
  script:
    - 'curl --header "Job-Token: $CI_JOB_TOKEN" --data tag=$CI_COMMIT_TAG "${CI_API_V4_URL}/projects/$CI_PROJECT_ID/packages/composer"'
  rules:
    - if: $CI_COMMIT_TAG
      allow_failure: true
  cache:
    key: cacheDeploy
    paths:
      - vendor/
      - node_modules/

deploy_staging:
  image: ubuntu:latest
  stage: staging
  before_script:
    - command -v ssh-agent || ( apt-get update -y && apt-get install openssh-client curl
      -y )
    - eval $(ssh-agent -s)
    - echo "$SSH_PRIVATE_KEY" | tr -d '\r' | ssh-add -
    - mkdir -p ~/.ssh
    - chmod 700 ~/.ssh
    - echo -e "Host *\n\tStrictHostKeyChecking no\n\n" > ~/.ssh/config
    - touch ~/.ssh/known_hosts
    - echo "$KNOWN_HOST" > ~/.ssh/known_hosts
  script:
    - ssh $SERVER "sh deploy.sh; exit;"
  rules:
    - if: '$CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH'
      allow_failure: true
  cache:
    key: cacheDeploy
    paths:
      - vendor/
      - node_modules/

browser_performance:
  variables:
    URL: "$DAST_WEBSITE"

release_job:
  stage: release
  image: registry.gitlab.com/gitlab-org/release-cli:latest
  rules:
    - if: $CI_COMMIT_TAG
  script:
    - echo 'running release_job'
  release:
    name: 'Release $CI_COMMIT_TAG'
    description: 'Created using the release-cli $EXTRA_DESCRIPTION'
    tag_name: '$CI_COMMIT_TAG'
    ref: '$CI_COMMIT_TAG'

include:
  - template: Jobs/Build.gitlab-ci.yml
  - template: Jobs/Test.gitlab-ci.yml
  - template: Jobs/Code-Quality.gitlab-ci.yml
  - template: Jobs/Code-Intelligence.gitlab-ci.yml
  - template: Jobs/DAST-Default-Branch-Deploy.gitlab-ci.yml
  - template: Verify/Browser-Performance.gitlab-ci.yml
  - template: Security/DAST.gitlab-ci.yml
  - template: Security/Container-Scanning.gitlab-ci.yml
  - template: Security/Dependency-Scanning.gitlab-ci.yml
  - template: Security/License-Scanning.gitlab-ci.yml
  - template: Security/SAST.gitlab-ci.yml
  - template: Security/Secret-Detection.gitlab-ci.yml

On self hosted gitlab runner, it’s always failed like this:

Running with gitlab-runner 14.5.2 (e91107dd)
  on runner-name redacted
Preparing the "docker" executor
Using Docker executor with image registry.gitlab.com/gitlab-org/cluster-integration/auto-build-image:v1.0.0 ...
Starting service docker:20.10.6-dind ...
Pulling docker image docker:20.10.6-dind ...
Using docker image sha256:dc8c389414c80f3c6510d3690cd03c29fc99d66f58955f138248499a34186bfa for docker:20.10.6-dind with digest docker@sha256:aa1e1fab842a35f16cf6e8fcb82b77b29be46f6aa8cd0fe69db0908940256be7 ...
Waiting for services to be up and running...
*** WARNING: Service runner-2icfbnnk-project-4-concurrent-0-d1bfdfee2f36d746-docker-0 probably didn't start properly.
Health check error:
service "runner-2icfbnnk-project-4-concurrent-0-d1bfdfee2f36d746-docker-0-wait-for-service" timeout
Health check container logs:
Service container logs:
2021-12-13T06:24:46.379513966Z time="2021-12-13T06:24:46.379345800Z" level=info msg="Starting up"
2021-12-13T06:24:46.380618763Z time="2021-12-13T06:24:46.380561315Z" level=warning msg="could not change group /var/run/docker.sock to docker: group docker not found"
2021-12-13T06:24:46.380755662Z time="2021-12-13T06:24:46.380707528Z" level=warning msg="Binding to IP address without --tlsverify is insecure and gives root access on this machine to everyone who has access to your network." host="tcp://0.0.0.0:2375"
2021-12-13T06:24:46.380770546Z time="2021-12-13T06:24:46.380729273Z" level=warning msg="Binding to an IP address, even on localhost, can also give access to scripts run in a browser. Be safe out there!" host="tcp://0.0.0.0:2375"
2021-12-13T06:24:47.383973972Z time="2021-12-13T06:24:47.383736399Z" level=info msg="libcontainerd: started new containerd process" pid=28
2021-12-13T06:24:47.384040042Z time="2021-12-13T06:24:47.383863614Z" level=info msg="parsed scheme: \"unix\"" module=grpc
2021-12-13T06:24:47.384069198Z time="2021-12-13T06:24:47.383901100Z" level=info msg="scheme \"unix\" not registered, fallback to default scheme" module=grpc
2021-12-13T06:24:47.384091684Z time="2021-12-13T06:24:47.383973716Z" level=info msg="ccResolverWrapper: sending update to cc: {[{unix:///var/run/docker/containerd/containerd.sock  <nil> 0 <nil>}] <nil> <nil>}" module=grpc
2021-12-13T06:24:47.384139233Z time="2021-12-13T06:24:47.384026005Z" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
2021-12-13T06:24:47.402294658Z time="2021-12-13T06:24:47.402193728Z" level=info msg="starting containerd" revision=05f951a3781f4f2c1911b05e61c160e9c30eaa8e version=v1.4.4
2021-12-13T06:24:47.422317352Z time="2021-12-13T06:24:47.422232485Z" level=info msg="loading plugin \"io.containerd.content.v1.content\"..." type=io.containerd.content.v1
2021-12-13T06:24:47.422467196Z time="2021-12-13T06:24:47.422421338Z" level=info msg="loading plugin \"io.containerd.snapshotter.v1.aufs\"..." type=io.containerd.snapshotter.v1
2021-12-13T06:24:47.688962816Z time="2021-12-13T06:24:47.688829387Z" level=info msg="loading plugin \"io.containerd.snapshotter.v1.btrfs\"..." type=io.containerd.snapshotter.v1
2021-12-13T06:24:47.689188014Z time="2021-12-13T06:24:47.689131210Z" level=info msg="skip loading plugin \"io.containerd.snapshotter.v1.btrfs\"..." error="path /var/lib/docker/containerd/daemon/io.containerd.snapshotter.v1.btrfs (ext4) must be a btrfs filesystem to be used with the btrfs snapshotter: skip plugin" type=io.containerd.snapshotter.v1
2021-12-13T06:24:47.689200985Z time="2021-12-13T06:24:47.689156326Z" level=info msg="loading plugin \"io.containerd.snapshotter.v1.devmapper\"..." type=io.containerd.snapshotter.v1
2021-12-13T06:24:47.689208702Z time="2021-12-13T06:24:47.689179846Z" level=warning msg="failed to load plugin io.containerd.snapshotter.v1.devmapper" error="devmapper not configured"
2021-12-13T06:24:47.689215962Z time="2021-12-13T06:24:47.689192480Z" level=info msg="loading plugin \"io.containerd.snapshotter.v1.native\"..." type=io.containerd.snapshotter.v1
2021-12-13T06:24:47.689310532Z time="2021-12-13T06:24:47.689268205Z" level=info msg="loading plugin \"io.containerd.snapshotter.v1.overlayfs\"..." type=io.containerd.snapshotter.v1
2021-12-13T06:24:47.689446138Z time="2021-12-13T06:24:47.689409101Z" level=info msg="loading plugin \"io.containerd.snapshotter.v1.zfs\"..." type=io.containerd.snapshotter.v1
2021-12-13T06:24:47.689674013Z time="2021-12-13T06:24:47.689608089Z" level=info msg="skip loading plugin \"io.containerd.snapshotter.v1.zfs\"..." error="path /var/lib/docker/containerd/daemon/io.containerd.snapshotter.v1.zfs must be a zfs filesystem to be used with the zfs snapshotter: skip plugin" type=io.containerd.snapshotter.v1
2021-12-13T06:24:47.689697746Z time="2021-12-13T06:24:47.689635938Z" level=info msg="loading plugin \"io.containerd.metadata.v1.bolt\"..." type=io.containerd.metadata.v1
2021-12-13T06:24:47.689737379Z time="2021-12-13T06:24:47.689704248Z" level=warning msg="could not use snapshotter devmapper in metadata plugin" error="devmapper not configured"
2021-12-13T06:24:47.689744992Z time="2021-12-13T06:24:47.689717500Z" level=info msg="metadata content store policy set" policy=shared
2021-12-13T06:24:48.220186692Z time="2021-12-13T06:24:48.219913859Z" level=info msg="loading plugin \"io.containerd.differ.v1.walking\"..." type=io.containerd.differ.v1
2021-12-13T06:24:48.220258318Z time="2021-12-13T06:24:48.220020534Z" level=info msg="loading plugin \"io.containerd.gc.v1.scheduler\"..." type=io.containerd.gc.v1
2021-12-13T06:24:48.220340594Z time="2021-12-13T06:24:48.220138403Z" level=info msg="loading plugin \"io.containerd.service.v1.introspection-service\"..." type=io.containerd.service.v1
2021-12-13T06:24:48.220482782Z time="2021-12-13T06:24:48.220290307Z" level=info msg="loading plugin \"io.containerd.service.v1.containers-service\"..." type=io.containerd.service.v1
2021-12-13T06:24:48.220519818Z time="2021-12-13T06:24:48.220401756Z" level=info msg="loading plugin \"io.containerd.service.v1.content-service\"..." type=io.containerd.service.v1
2021-12-13T06:24:48.220614980Z time="2021-12-13T06:24:48.220474202Z" level=info msg="loading plugin \"io.containerd.service.v1.diff-service\"..." type=io.containerd.service.v1
2021-12-13T06:24:48.220664742Z time="2021-12-13T06:24:48.220567301Z" level=info msg="loading plugin \"io.containerd.service.v1.images-service\"..." type=io.containerd.service.v1
2021-12-13T06:24:48.220742715Z time="2021-12-13T06:24:48.220642160Z" level=info msg="loading plugin \"io.containerd.service.v1.leases-service\"..." type=io.containerd.service.v1
2021-12-13T06:24:48.220883066Z time="2021-12-13T06:24:48.220721418Z" level=info msg="loading plugin \"io.containerd.service.v1.namespaces-service\"..." type=io.containerd.service.v1
2021-12-13T06:24:48.221002245Z time="2021-12-13T06:24:48.220796041Z" level=info msg="loading plugin \"io.containerd.service.v1.snapshots-service\"..." type=io.containerd.service.v1
2021-12-13T06:24:48.221031228Z time="2021-12-13T06:24:48.220871156Z" level=info msg="loading plugin \"io.containerd.runtime.v1.linux\"..." type=io.containerd.runtime.v1
2021-12-13T06:24:48.221507618Z time="2021-12-13T06:24:48.221379836Z" level=info msg="loading plugin \"io.containerd.runtime.v2.task\"..." type=io.containerd.runtime.v2
2021-12-13T06:24:48.221823767Z time="2021-12-13T06:24:48.221698785Z" level=info msg="loading plugin \"io.containerd.monitor.v1.cgroups\"..." type=io.containerd.monitor.v1
2021-12-13T06:24:48.222805024Z time="2021-12-13T06:24:48.222651793Z" level=info msg="loading plugin \"io.containerd.service.v1.tasks-service\"..." type=io.containerd.service.v1
2021-12-13T06:24:48.222848066Z time="2021-12-13T06:24:48.222722418Z" level=info msg="loading plugin \"io.containerd.internal.v1.restart\"..." type=io.containerd.internal.v1
2021-12-13T06:24:48.222935869Z time="2021-12-13T06:24:48.222836103Z" level=info msg="loading plugin \"io.containerd.grpc.v1.containers\"..." type=io.containerd.grpc.v1
2021-12-13T06:24:48.222966287Z time="2021-12-13T06:24:48.222916453Z" level=info msg="loading plugin \"io.containerd.grpc.v1.content\"..." type=io.containerd.grpc.v1
2021-12-13T06:24:48.222977573Z time="2021-12-13T06:24:48.222943287Z" level=info msg="loading plugin \"io.containerd.grpc.v1.diff\"..." type=io.containerd.grpc.v1
2021-12-13T06:24:48.223003602Z time="2021-12-13T06:24:48.222960901Z" level=info msg="loading plugin \"io.containerd.grpc.v1.events\"..." type=io.containerd.grpc.v1
2021-12-13T06:24:48.223014536Z time="2021-12-13T06:24:48.222980315Z" level=info msg="loading plugin \"io.containerd.grpc.v1.healthcheck\"..." type=io.containerd.grpc.v1
2021-12-13T06:24:48.223043047Z time="2021-12-13T06:24:48.222999596Z" level=info msg="loading plugin \"io.containerd.grpc.v1.images\"..." type=io.containerd.grpc.v1
2021-12-13T06:24:48.223052368Z time="2021-12-13T06:24:48.223025666Z" level=info msg="loading plugin \"io.containerd.grpc.v1.leases\"..." type=io.containerd.grpc.v1
2021-12-13T06:24:48.223070659Z time="2021-12-13T06:24:48.223041267Z" level=info msg="loading plugin \"io.containerd.grpc.v1.namespaces\"..." type=io.containerd.grpc.v1
2021-12-13T06:24:48.223087893Z time="2021-12-13T06:24:48.223058204Z" level=info msg="loading plugin \"io.containerd.internal.v1.opt\"..." type=io.containerd.internal.v1
2021-12-13T06:24:48.223312969Z time="2021-12-13T06:24:48.223266471Z" level=info msg="loading plugin \"io.containerd.grpc.v1.snapshots\"..." type=io.containerd.grpc.v1
2021-12-13T06:24:48.223321547Z time="2021-12-13T06:24:48.223291256Z" level=info msg="loading plugin \"io.containerd.grpc.v1.tasks\"..." type=io.containerd.grpc.v1
2021-12-13T06:24:48.223334940Z time="2021-12-13T06:24:48.223307760Z" level=info msg="loading plugin \"io.containerd.grpc.v1.version\"..." type=io.containerd.grpc.v1
2021-12-13T06:24:48.223350478Z time="2021-12-13T06:24:48.223327029Z" level=info msg="loading plugin \"io.containerd.grpc.v1.introspection\"..." type=io.containerd.grpc.v1
2021-12-13T06:24:48.223610658Z time="2021-12-13T06:24:48.223553624Z" level=info msg=serving... address=/var/run/docker/containerd/containerd-debug.sock
2021-12-13T06:24:48.223691657Z time="2021-12-13T06:24:48.223643418Z" level=info msg=serving... address=/var/run/docker/containerd/containerd.sock.ttrpc
2021-12-13T06:24:48.223753892Z time="2021-12-13T06:24:48.223718639Z" level=info msg=serving... address=/var/run/docker/containerd/containerd.sock
2021-12-13T06:24:48.223789287Z time="2021-12-13T06:24:48.223744929Z" level=info msg="containerd successfully booted in 0.822209s"
2021-12-13T06:24:48.230210427Z time="2021-12-13T06:24:48.230142530Z" level=info msg="Setting the storage driver from the $DOCKER_DRIVER environment variable (overlay2)"
2021-12-13T06:24:48.230378069Z time="2021-12-13T06:24:48.230327026Z" level=info msg="parsed scheme: \"unix\"" module=grpc
2021-12-13T06:24:48.230388165Z time="2021-12-13T06:24:48.230344584Z" level=info msg="scheme \"unix\" not registered, fallback to default scheme" module=grpc
2021-12-13T06:24:48.230405242Z time="2021-12-13T06:24:48.230369476Z" level=info msg="ccResolverWrapper: sending update to cc: {[{unix:///var/run/docker/containerd/containerd.sock  <nil> 0 <nil>}] <nil> <nil>}" module=grpc
2021-12-13T06:24:48.230415216Z time="2021-12-13T06:24:48.230387459Z" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
2021-12-13T06:24:48.231056099Z time="2021-12-13T06:24:48.231001784Z" level=info msg="parsed scheme: \"unix\"" module=grpc
2021-12-13T06:24:48.231069665Z time="2021-12-13T06:24:48.231019580Z" level=info msg="scheme \"unix\" not registered, fallback to default scheme" module=grpc
2021-12-13T06:24:48.231087522Z time="2021-12-13T06:24:48.231035665Z" level=info msg="ccResolverWrapper: sending update to cc: {[{unix:///var/run/docker/containerd/containerd.sock  <nil> 0 <nil>}] <nil> <nil>}" module=grpc
2021-12-13T06:24:48.231097702Z time="2021-12-13T06:24:48.231055115Z" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
2021-12-13T06:24:48.957801275Z time="2021-12-13T06:24:48.957578721Z" level=warning msg="Your kernel does not support swap memory limit"
2021-12-13T06:24:48.957860031Z time="2021-12-13T06:24:48.957644920Z" level=warning msg="Your kernel does not support CPU realtime scheduler"
2021-12-13T06:24:48.957886607Z time="2021-12-13T06:24:48.957670963Z" level=warning msg="Your kernel does not support cgroup blkio weight"
2021-12-13T06:24:48.957904334Z time="2021-12-13T06:24:48.957696187Z" level=warning msg="Your kernel does not support cgroup blkio weight_device"
2021-12-13T06:24:48.958433107Z time="2021-12-13T06:24:48.958299182Z" level=info msg="Loading containers: start."
2021-12-13T06:24:49.472685296Z time="2021-12-13T06:24:49.472558476Z" level=info msg="Default bridge (docker0) is assigned with an IP address 172.18.0.0/16. Daemon option --bip can be used to set a preferred IP address"
2021-12-13T06:24:49.705270455Z time="2021-12-13T06:24:49.705032558Z" level=info msg="Loading containers: done."
2021-12-13T06:24:50.359460942Z time="2021-12-13T06:24:50.359261713Z" level=info msg="Docker daemon" commit=8728dd2 graphdriver(s)=overlay2 version=20.10.6
2021-12-13T06:24:50.359693692Z time="2021-12-13T06:24:50.359582579Z" level=info msg="Daemon has completed initialization"
2021-12-13T06:24:50.835925240Z time="2021-12-13T06:24:50.835801298Z" level=info msg="API listen on /var/run/docker.sock"
2021-12-13T06:24:50.843313950Z time="2021-12-13T06:24:50.843190759Z" level=info msg="API listen on [::]:2375"
*********
Pulling docker image registry.gitlab.com/gitlab-org/cluster-integration/auto-build-image:v1.0.0 ...
Using docker image sha256:32e6c034cd671a75d3c39f5523317b1278ede2325bdd248d4bc6152e3b9c6bba for registry.gitlab.com/gitlab-org/cluster-integration/auto-build-image:v1.0.0 with digest registry.gitlab.com/gitlab-org/cluster-integration/auto-build-image@sha256:80cf4f25f26d13027bbc0ab12cb475d49d13ab8b24144414caaf6476ef956a67 ...
Preparing environment
Running on runner-2icfbnnk-project-4-concurrent-0 via 9081563381c4...
Getting source from Git repository
Fetching changes with git depth set to 50...
Initialized empty Git repository in /builds/groups/project/.git/
Created fresh repository.
Checking out 60821a63 as some-branch...
Skipping Git submodules setup
Executing "step_script" stage of the job script
Using docker image sha256:32e6c034cd671a75d3c39f5523317b1278ede2325bdd248d4bc6152e3b9c6bba for registry.gitlab.com/gitlab-org/cluster-integration/auto-build-image:v1.0.0 with digest registry.gitlab.com/gitlab-org/cluster-integration/auto-build-image@sha256:80cf4f25f26d13027bbc0ab12cb475d49d13ab8b24144414caaf6476ef956a67 ...
$ if [[ -z "$CI_COMMIT_TAG" ]]; then # collapsed multi-line command
$ /build/build.sh
Building Dockerfile-based application...
Attempting to pull a previously built image for use with --cache-from...
invalid reference format
invalid reference format
No previously cached image found. The docker build will proceed without using a cached image
invalid argument "/some-branch:60821a633960e52da81bddd65d8983588c3fc657" for "-t, --tag" flag: invalid reference format
See 'docker build --help'.
ERROR: Job failed: exit code 1

But the above job always success on gitlab.com.

Is there something wrong on my gitlab runner configuration?

1 Like
2

Hi @asdasdasd

According to this you need:

  [runners.docker]
    tls_verify = false
    image = "ubuntu:latest"
    privileged = true
    disable_cache = false
    volumes = ["/certs/client", "/cache"]
    shm_size = 0

You might also want to add pull_policy = "if-not-present" to avoid pulling images that you already have on local disk.

3

Hi @snim2, thank you for the response.

I have followed your post and changed the runners config to be like this:


concurrent = 1
check_interval = 0

[session_server]
  session_timeout = 1800

[[runners]]
  name = "gitlabrunner"
  url = "http://gitlab-domain-name/"
  token = "redacted"
  executor = "docker"
  [runners.custom_build_dir]
  [runners.cache]
    [runners.cache.s3]
    [runners.cache.gcs]
    [runners.cache.azure]
  [runners.docker]
    tls_verify = false
    image = "ubuntu:latest"
    privileged = true
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = false
    volumes = ["/certs/client", "/cache"]
    shm_size = 0

I edited from the host server, and then rebuild the docker with this oneliner command:

docker pull gitlab/gitlab-runner:latest && docker stop gitlab-runner && docker rm gitlab-runner && docker run -d --name gitlab-runner --restart always -v /var/run/docker.sock:/var/run/docker.sock -v /path/to/gitlab-runner/on-host/config:/etc/gitlab-runner gitlab/gitlab-runner:latest

after the gitlab-runner docker is up, I checked the configuration via:

docker exec -it gitlab-runner /bin/bash

and
$ gitlab-runner list
$ gitlab-runner verify

it shows that my runner is configured.

But when I try to re-run the failed job, it’s still failed with error like this:


Running with gitlab-runner 14.5.2 (commit-id)
  on runner-name redacted
Preparing the "docker" executor
Using Docker executor with image registry.gitlab.com/gitlab-org/cluster-integration/auto-build-image:v1.0.0 ...
Starting service docker:20.10.6-dind ...
Pulling docker image docker:20.10.6-dind ...
Using docker image sha256:dc8c389414c80f3c6510d3690cd03c29fc99d66f58955f138248499a34186bfa for docker:20.10.6-dind with digest docker@sha256:aa1e1fab842a35f16cf6e8fcb82b77b29be46f6aa8cd0fe69db0908940256be7 ...
Waiting for services to be up and running...
*** WARNING: Service runner-2icfbnnk-project-4-concurrent-0-ce2c52fb1938afbc-docker-0 probably didn't start properly.
Health check error:
service "runner-2icfbnnk-project-4-concurrent-0-ce2c52fb1938afbc-docker-0-wait-for-service" timeout
Health check container logs:
Service container logs:
2021-12-17T06:59:06.608404149Z time="2021-12-17T06:59:06.608213798Z" level=info msg="Starting up"
2021-12-17T06:59:06.609699912Z time="2021-12-17T06:59:06.609624295Z" level=warning msg="could not change group /var/run/docker.sock to docker: group docker not found"
2021-12-17T06:59:06.609907543Z time="2021-12-17T06:59:06.609827972Z" level=warning msg="Binding to IP address without --tlsverify is insecure and gives root access on this machine to everyone who has access to your network." host="tcp://0.0.0.0:2375"
2021-12-17T06:59:06.609929759Z time="2021-12-17T06:59:06.609861083Z" level=warning msg="Binding to an IP address, even on localhost, can also give access to scripts run in a browser. Be safe out there!" host="tcp://0.0.0.0:2375"
2021-12-17T06:59:07.611205604Z time="2021-12-17T06:59:07.611082974Z" level=info msg="libcontainerd: started new containerd process" pid=26
2021-12-17T06:59:07.611228476Z time="2021-12-17T06:59:07.611136064Z" level=info msg="parsed scheme: \"unix\"" module=grpc
2021-12-17T06:59:07.611236286Z time="2021-12-17T06:59:07.611150689Z" level=info msg="scheme \"unix\" not registered, fallback to default scheme" module=grpc
2021-12-17T06:59:07.611243360Z time="2021-12-17T06:59:07.611180338Z" level=info msg="ccResolverWrapper: sending update to cc: {[{unix:///var/run/docker/containerd/containerd.sock  <nil> 0 <nil>}] <nil> <nil>}" module=grpc
2021-12-17T06:59:07.611248096Z time="2021-12-17T06:59:07.611198300Z" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
2021-12-17T06:59:07.689553631Z time="2021-12-17T06:59:07.689435724Z" level=info msg="starting containerd" revision=05f951a3781f4f2c1911b05e61c160e9c30eaa8e version=v1.4.4
2021-12-17T06:59:07.709831858Z time="2021-12-17T06:59:07.709683858Z" level=info msg="loading plugin \"io.containerd.content.v1.content\"..." type=io.containerd.content.v1
2021-12-17T06:59:07.709980117Z time="2021-12-17T06:59:07.709935840Z" level=info msg="loading plugin \"io.containerd.snapshotter.v1.aufs\"..." type=io.containerd.snapshotter.v1
2021-12-17T06:59:07.780415650Z time="2021-12-17T06:59:07.780317524Z" level=info msg="loading plugin \"io.containerd.snapshotter.v1.btrfs\"..." type=io.containerd.snapshotter.v1
2021-12-17T06:59:07.780609182Z time="2021-12-17T06:59:07.780555690Z" level=info msg="skip loading plugin \"io.containerd.snapshotter.v1.btrfs\"..." error="path /var/lib/docker/containerd/daemon/io.containerd.snapshotter.v1.btrfs (ext4) must be a btrfs filesystem to be used with the btrfs snapshotter: skip plugin" type=io.containerd.snapshotter.v1
2021-12-17T06:59:07.780622173Z time="2021-12-17T06:59:07.780579936Z" level=info msg="loading plugin \"io.containerd.snapshotter.v1.devmapper\"..." type=io.containerd.snapshotter.v1
2021-12-17T06:59:07.780628267Z time="2021-12-17T06:59:07.780597765Z" level=warning msg="failed to load plugin io.containerd.snapshotter.v1.devmapper" error="devmapper not configured"
2021-12-17T06:59:07.780632384Z time="2021-12-17T06:59:07.780609854Z" level=info msg="loading plugin \"io.containerd.snapshotter.v1.native\"..." type=io.containerd.snapshotter.v1
2021-12-17T06:59:07.780694188Z time="2021-12-17T06:59:07.780659607Z" level=info msg="loading plugin \"io.containerd.snapshotter.v1.overlayfs\"..." type=io.containerd.snapshotter.v1
2021-12-17T06:59:07.780830318Z time="2021-12-17T06:59:07.780780899Z" level=info msg="loading plugin \"io.containerd.snapshotter.v1.zfs\"..." type=io.containerd.snapshotter.v1
2021-12-17T06:59:07.780975868Z time="2021-12-17T06:59:07.780933377Z" level=info msg="skip loading plugin \"io.containerd.snapshotter.v1.zfs\"..." error="path /var/lib/docker/containerd/daemon/io.containerd.snapshotter.v1.zfs must be a zfs filesystem to be used with the zfs snapshotter: skip plugin" type=io.containerd.snapshotter.v1
2021-12-17T06:59:07.780983744Z time="2021-12-17T06:59:07.780950852Z" level=info msg="loading plugin \"io.containerd.metadata.v1.bolt\"..." type=io.containerd.metadata.v1
2021-12-17T06:59:07.781023387Z time="2021-12-17T06:59:07.780989497Z" level=warning msg="could not use snapshotter devmapper in metadata plugin" error="devmapper not configured"
2021-12-17T06:59:07.781033284Z time="2021-12-17T06:59:07.781003385Z" level=info msg="metadata content store policy set" policy=shared
2021-12-17T06:59:08.043668954Z time="2021-12-17T06:59:08.043414721Z" level=info msg="loading plugin \"io.containerd.differ.v1.walking\"..." type=io.containerd.differ.v1
2021-12-17T06:59:08.043724140Z time="2021-12-17T06:59:08.043499213Z" level=info msg="loading plugin \"io.containerd.gc.v1.scheduler\"..." type=io.containerd.gc.v1
2021-12-17T06:59:08.043738085Z time="2021-12-17T06:59:08.043580224Z" level=info msg="loading plugin \"io.containerd.service.v1.introspection-service\"..." type=io.containerd.service.v1
2021-12-17T06:59:08.043837789Z time="2021-12-17T06:59:08.043680151Z" level=info msg="loading plugin \"io.containerd.service.v1.containers-service\"..." type=io.containerd.service.v1
2021-12-17T06:59:08.043881318Z time="2021-12-17T06:59:08.043754639Z" level=info msg="loading plugin \"io.containerd.service.v1.content-service\"..." type=io.containerd.service.v1
2021-12-17T06:59:08.043908034Z time="2021-12-17T06:59:08.043817113Z" level=info msg="loading plugin \"io.containerd.service.v1.diff-service\"..." type=io.containerd.service.v1
2021-12-17T06:59:08.044024453Z time="2021-12-17T06:59:08.043894165Z" level=info msg="loading plugin \"io.containerd.service.v1.images-service\"..." type=io.containerd.service.v1
2021-12-17T06:59:08.044057583Z time="2021-12-17T06:59:08.043964948Z" level=info msg="loading plugin \"io.containerd.service.v1.leases-service\"..." type=io.containerd.service.v1
2021-12-17T06:59:08.044108480Z time="2021-12-17T06:59:08.044012110Z" level=info msg="loading plugin \"io.containerd.service.v1.namespaces-service\"..." type=io.containerd.service.v1
2021-12-17T06:59:08.044191099Z time="2021-12-17T06:59:08.044063266Z" level=info msg="loading plugin \"io.containerd.service.v1.snapshots-service\"..." type=io.containerd.service.v1
2021-12-17T06:59:08.044239589Z time="2021-12-17T06:59:08.044152761Z" level=info msg="loading plugin \"io.containerd.runtime.v1.linux\"..." type=io.containerd.runtime.v1
2021-12-17T06:59:08.044750596Z time="2021-12-17T06:59:08.044628465Z" level=info msg="loading plugin \"io.containerd.runtime.v2.task\"..." type=io.containerd.runtime.v2
2021-12-17T06:59:08.045065067Z time="2021-12-17T06:59:08.044974417Z" level=info msg="loading plugin \"io.containerd.monitor.v1.cgroups\"..." type=io.containerd.monitor.v1
2021-12-17T06:59:08.045939374Z time="2021-12-17T06:59:08.045793711Z" level=info msg="loading plugin \"io.containerd.service.v1.tasks-service\"..." type=io.containerd.service.v1
2021-12-17T06:59:08.045970866Z time="2021-12-17T06:59:08.045875258Z" level=info msg="loading plugin \"io.containerd.internal.v1.restart\"..." type=io.containerd.internal.v1
2021-12-17T06:59:08.046180815Z time="2021-12-17T06:59:08.046041157Z" level=info msg="loading plugin \"io.containerd.grpc.v1.containers\"..." type=io.containerd.grpc.v1
2021-12-17T06:59:08.046203431Z time="2021-12-17T06:59:08.046104076Z" level=info msg="loading plugin \"io.containerd.grpc.v1.content\"..." type=io.containerd.grpc.v1
2021-12-17T06:59:08.046270794Z time="2021-12-17T06:59:08.046167423Z" level=info msg="loading plugin \"io.containerd.grpc.v1.diff\"..." type=io.containerd.grpc.v1
2021-12-17T06:59:08.046320595Z time="2021-12-17T06:59:08.046229465Z" level=info msg="loading plugin \"io.containerd.grpc.v1.events\"..." type=io.containerd.grpc.v1
2021-12-17T06:59:08.046413977Z time="2021-12-17T06:59:08.046296016Z" level=info msg="loading plugin \"io.containerd.grpc.v1.healthcheck\"..." type=io.containerd.grpc.v1
2021-12-17T06:59:08.046446071Z time="2021-12-17T06:59:08.046354147Z" level=info msg="loading plugin \"io.containerd.grpc.v1.images\"..." type=io.containerd.grpc.v1
2021-12-17T06:59:08.046509285Z time="2021-12-17T06:59:08.046409041Z" level=info msg="loading plugin \"io.containerd.grpc.v1.leases\"..." type=io.containerd.grpc.v1
2021-12-17T06:59:08.046582018Z time="2021-12-17T06:59:08.046486782Z" level=info msg="loading plugin \"io.containerd.grpc.v1.namespaces\"..." type=io.containerd.grpc.v1
2021-12-17T06:59:08.046659151Z time="2021-12-17T06:59:08.046553104Z" level=info msg="loading plugin \"io.containerd.internal.v1.opt\"..." type=io.containerd.internal.v1
2021-12-17T06:59:08.047127219Z time="2021-12-17T06:59:08.046993129Z" level=info msg="loading plugin \"io.containerd.grpc.v1.snapshots\"..." type=io.containerd.grpc.v1
2021-12-17T06:59:08.047143633Z time="2021-12-17T06:59:08.047053909Z" level=info msg="loading plugin \"io.containerd.grpc.v1.tasks\"..." type=io.containerd.grpc.v1
2021-12-17T06:59:08.047201509Z time="2021-12-17T06:59:08.047117823Z" level=info msg="loading plugin \"io.containerd.grpc.v1.version\"..." type=io.containerd.grpc.v1
2021-12-17T06:59:08.047253115Z time="2021-12-17T06:59:08.047183816Z" level=info msg="loading plugin \"io.containerd.grpc.v1.introspection\"..." type=io.containerd.grpc.v1
2021-12-17T06:59:08.047702534Z time="2021-12-17T06:59:08.047596579Z" level=info msg=serving... address=/var/run/docker/containerd/containerd-debug.sock
2021-12-17T06:59:08.047870891Z time="2021-12-17T06:59:08.047763346Z" level=info msg=serving... address=/var/run/docker/containerd/containerd.sock.ttrpc
2021-12-17T06:59:08.048036150Z time="2021-12-17T06:59:08.047917631Z" level=info msg=serving... address=/var/run/docker/containerd/containerd.sock
2021-12-17T06:59:08.048183127Z time="2021-12-17T06:59:08.048010422Z" level=info msg="containerd successfully booted in 0.383533s"
2021-12-17T06:59:08.056777653Z time="2021-12-17T06:59:08.056676146Z" level=info msg="Setting the storage driver from the $DOCKER_DRIVER environment variable (overlay2)"
2021-12-17T06:59:08.057016830Z time="2021-12-17T06:59:08.056933811Z" level=info msg="parsed scheme: \"unix\"" module=grpc
2021-12-17T06:59:08.057041505Z time="2021-12-17T06:59:08.056957276Z" level=info msg="scheme \"unix\" not registered, fallback to default scheme" module=grpc
2021-12-17T06:59:08.057049097Z time="2021-12-17T06:59:08.056981019Z" level=info msg="ccResolverWrapper: sending update to cc: {[{unix:///var/run/docker/containerd/containerd.sock  <nil> 0 <nil>}] <nil> <nil>}" module=grpc
2021-12-17T06:59:08.057055745Z time="2021-12-17T06:59:08.056996986Z" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
2021-12-17T06:59:08.057888184Z time="2021-12-17T06:59:08.057808346Z" level=info msg="parsed scheme: \"unix\"" module=grpc
2021-12-17T06:59:08.057904650Z time="2021-12-17T06:59:08.057831084Z" level=info msg="scheme \"unix\" not registered, fallback to default scheme" module=grpc
2021-12-17T06:59:08.057911445Z time="2021-12-17T06:59:08.057849994Z" level=info msg="ccResolverWrapper: sending update to cc: {[{unix:///var/run/docker/containerd/containerd.sock  <nil> 0 <nil>}] <nil> <nil>}" module=grpc
2021-12-17T06:59:08.057919455Z time="2021-12-17T06:59:08.057866466Z" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
2021-12-17T06:59:09.547918322Z time="2021-12-17T06:59:09.547803452Z" level=warning msg="Your kernel does not support swap memory limit"
2021-12-17T06:59:09.547950816Z time="2021-12-17T06:59:09.547835199Z" level=warning msg="Your kernel does not support CPU realtime scheduler"
2021-12-17T06:59:09.547956219Z time="2021-12-17T06:59:09.547844448Z" level=warning msg="Your kernel does not support cgroup blkio weight"
2021-12-17T06:59:09.547959894Z time="2021-12-17T06:59:09.547853707Z" level=warning msg="Your kernel does not support cgroup blkio weight_device"
2021-12-17T06:59:09.548120614Z time="2021-12-17T06:59:09.548063732Z" level=info msg="Loading containers: start."
2021-12-17T06:59:10.082166650Z time="2021-12-17T06:59:10.082016699Z" level=info msg="Default bridge (docker0) is assigned with an IP address 172.18.0.0/16. Daemon option --bip can be used to set a preferred IP address"
2021-12-17T06:59:10.248673504Z time="2021-12-17T06:59:10.248559942Z" level=info msg="Loading containers: done."
2021-12-17T06:59:10.931199982Z time="2021-12-17T06:59:10.931067356Z" level=info msg="Docker daemon" commit=8728dd2 graphdriver(s)=overlay2 version=20.10.6
2021-12-17T06:59:10.931264805Z time="2021-12-17T06:59:10.931221351Z" level=info msg="Daemon has completed initialization"
2021-12-17T06:59:11.515846387Z time="2021-12-17T06:59:11.515692351Z" level=info msg="API listen on /var/run/docker.sock"
2021-12-17T06:59:11.522369795Z time="2021-12-17T06:59:11.522258678Z" level=info msg="API listen on [::]:2375"
*********
Pulling docker image registry.gitlab.com/gitlab-org/cluster-integration/auto-build-image:v1.0.0 ...
Using docker image sha256:32e6c034cd671a75d3c39f5523317b1278ede2325bdd248d4bc6152e3b9c6bba for registry.gitlab.com/gitlab-org/cluster-integration/auto-build-image:v1.0.0 with digest registry.gitlab.com/gitlab-org/cluster-integration/auto-build-image@sha256:80cf4f25f26d13027bbc0ab12cb475d49d13ab8b24144414caaf6476ef956a67 ...
Preparing environment
00:06
Running on runner-2icfbnnk-project-4-concurrent-0 via b7a999c730af...
Getting source from Git repository
00:09
Fetching changes with git depth set to 50...
Reinitialized existing Git repository in /builds/groups-name/project/.git/
Checking out 60821a63 as some-branch...
Skipping Git submodules setup
Executing "step_script" stage of the job script
Using docker image sha256:32e6c034cd671a75d3c39f5523317b1278ede2325bdd248d4bc6152e3b9c6bba for registry.gitlab.com/gitlab-org/cluster-integration/auto-build-image:v1.0.0 with digest registry.gitlab.com/gitlab-org/cluster-integration/auto-build-image@sha256:80cf4f25f26d13027bbc0ab12cb475d49d13ab8b24144414caaf6476ef956a67 ...
$ if [[ -z "$CI_COMMIT_TAG" ]]; then # collapsed multi-line command
$ /build/build.sh
Building Dockerfile-based application...
Attempting to pull a previously built image for use with --cache-from...
invalid reference format
invalid reference format
No previously cached image found. The docker build will proceed without using a cached image
invalid argument "/some-branch:60821a633960e52da81bddd65d8983588c3fc657" for "-t, --tag" flag: invalid reference format
See 'docker build --help'.
ERROR: Job failed: exit code 1

Is there still something I missed? Because on gitlab.com runner, it’s work well. Thank you.

4

Hi there,

I guess you installed gitlab-runner like this? Usually I install it as as systemd service, so I’m not sure if I’m the best person to help you!

However, looking through your logs, you have /var/run/docker.sock to docker: group docker not found, So I would start with that. Is there a group on your system called docker? If not, should there be, and is gitlab-runner running as a user that is in the docker group?

5

I am having the same problem. Did you ever figure it out?