Gitlab runner, cannot login to Gitlab registry, Docker executor (lookup registry.gitlab.com: no such host)

Problem to solve

I encounter this odd problem when I run a registered runner (Docker executor, Dind service) on one of my machines:

$ echo $CI_JOB_TOKEN | docker login -u $CI_REGISTRY_USER --password-stdin $CI_REGISTRY
Error response from daemon: Get "https://registry.gitlab.com/v2/": dial tcp: lookup registry.gitlab.com on 192.168.0.1:53: no such host

I expect Docker engine to correctly find the GitLab URL IP and connect to it.

Steps to reproduce

I’ve tried various combinations of setup, from advice given on similar problems. I parsed again the GitLab Runner Doc., the Docker executor section. Tried different versions of docker image+dind service.
Another oddity is that, when I try to use Dind with TLS (DOCKER_TLS_CERTDIR: "/certs") I get another error:

Cannot connect to the Docker daemon at tcp://docker:2375. Is the docker daemon running?

I am completely clueless. :man_shrugging:
I’ve introduced some extra scripts in before_script block, to bring some clarity, if it helps.

Configuration

My setup is as follows:

  • Runner:
    Docker executor ( privileged = true), image: docker:25.0.5, services: [docker:25.0.5-dind]
  • .gitlab-ci.yml
    • Variables:
    DOCKER_DRIVER: overlay2
    DOCKER_HOST: tcp://docker:2375
    DOCKER_TLS_CERTDIR: ""
    
  • before_script
    - echo "nameserver 8.8.8.8" >> /etc/resolv.conf
    - df
    - cat /etc/resolv.conf
    - cat /etc/hosts
    - nslookup registry.gitlab.com
    - docker version
    - docker info
    - echo $CI_JOB_TOKEN | docker login -u $CI_REGISTRY_USER --password-stdin $CI_REGISTRY
    

:warning: :spiral_notepad: The complete output I get (below):

Expand/collapse
Running with gitlab-runner 16.11.0 (91a27b2a)
  on proj-at-alex-tower xxxxxxxxxxxx, system ID: xxxxxxxxxxxxxxx
Preparing the "docker" executor 00:24
Using Docker executor with image docker:25.0.5 ...
Starting service docker:25.0.5-dind ...
Pulling docker image docker:25.0.5-dind ...
Using docker image sha256:7d98e21a344ca2174b2061d438961e7adb47a774a7c6885318302d1f96b5f1c2 for docker:25.0.5-dind with digest docker@sha256:ee15a0c3a0ba9cb66b2bec8ff684096c151f94cb3f85e50c22d843aafbc9acab ...
Waiting for services to be up and running (timeout 30 seconds)...
Pulling docker image docker:25.0.5 ...
Using docker image sha256:7d98e21a344ca2174b2061d438961e7adb47a774a7c6885318302d1f96b5f1c2 for docker:25.0.5 with digest docker@sha256:ee15a0c3a0ba9cb66b2bec8ff684096c151f94cb3f85e50c22d843aafbc9acab ...
Preparing environment 00:01
Running on runner-zsbazvte-project-7675967-concurrent-0 via 35890dce4753...
Getting source from Git repository 00:02
Fetching changes...
Reinitialized existing Git repository in /builds/busuioc-alexandru/proj/.git/
Checking out a229a7eb as detached HEAD (ref is feat/ci-setup)...
Skipping Git submodules setup
Executing "step_script" stage of the job script 00:02
Using docker image sha256:7d98e21a344ca2174b2061d438961e7adb47a774a7c6885318302d1f96b5f1c2 for docker:25.0.5 with digest docker@sha256:ee15a0c3a0ba9cb66b2bec8ff684096c151f94cb3f85e50c22d843aafbc9acab ...
$ echo "nameserver 8.8.8.8" >> /etc/resolv.conf
$ df
Filesystem           1K-blocks      Used Available Use% Mounted on
overlay              503399440  70554428 407200264  15% /
tmpfs                    65536         0     65536   0% /dev
shm                      65536         0     65536   0% /dev/shm
/dev/sda5            503399440  70554428 407200264  15% /cache
/dev/sda5            503399440  70554428 407200264  15% /builds
/dev/sda5            503399440  70554428 407200264  15% /certs/client
/dev/sda5            503399440  70554428 407200264  15% /etc/resolv.conf
/dev/sda5            503399440  70554428 407200264  15% /etc/hostname
/dev/sda5            503399440  70554428 407200264  15% /etc/hosts
/dev/sda5            503399440  70554428 407200264  15% /var/lib/docker
$ cat /etc/resolv.conf
# Generated by Docker Engine.
# This file can be edited; Docker Engine will not make further changes once it
# has been modified.
nameserver 192.168.0.1
search .
# Based on host file: '/run/systemd/resolve/resolv.conf' (legacy)
# Overrides: []
nameserver 8.8.8.8
$ cat /etc/hosts
127.0.0.1	localhost
::1	localhost ip6-localhost ip6-loopback
fe00::0	ip6-localnet
ff00::0	ip6-mcastprefix
ff02::1	ip6-allnodes
ff02::2	ip6-allrouters
172.17.0.2	docker 361d7006a874 runner-zsbazvte-project-7675967-concurrent-0-ed19dd2622929a58-docker-0
172.17.0.3	runner-zsbazvte-project-7675967-concurrent-0
$ nslookup registry.gitlab.com
Server:		192.168.0.1
Address:	192.168.0.1:53
Non-authoritative answer:
Name:	registry.gitlab.com
Address: 35.227.35.254
Non-authoritative answer:
$ docker version
Client:
 Version:           25.0.5
 API version:       1.44
 Go version:        go1.21.8
 Git commit:        5dc9bcc
 Built:             Tue Mar 19 15:04:17 2024
 OS/Arch:           linux/amd64
 Context:           default
Server: Docker Engine - Community
 Engine:
  Version:          25.0.5
  API version:      1.44 (minimum version 1.24)
  Go version:       go1.21.8
  Git commit:       e63daec
  Built:            Tue Mar 19 15:05:39 2024
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.7.13
  GitCommit:        7c3aca7a610df76212171d200ca3811ff6096eb8
 runc:
  Version:          1.1.12
  GitCommit:        v1.1.12-0-g51d5e94
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0
$ docker info
Client:
 Version:    25.0.5
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.14.0
    Path:     /usr/local/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.27.0
    Path:     /usr/local/libexec/docker/cli-plugins/docker-compose
Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 0
 Server Version: 25.0.5
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 7c3aca7a610df76212171d200ca3811ff6096eb8
 runc version: v1.1.12-0-g51d5e94
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 6.5.0-27-generic
 Operating System: Alpine Linux v3.19 (containerized)
 OSType: linux
 Architecture: x86_64
 CPUs: 8
 Total Memory: 46.91GiB
 Name: 361d7006a874
 ID: b0c17175-93d8-4f8e-a02c-0e227d8c7c5f
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false
 Product License: Community Engine
WARNING: API is accessible on http://0.0.0.0:2375 without encryption.
         Access to the remote API is equivalent to root access on the host. Refer
         to the 'Docker daemon attack surface' section in the documentation for
         more information: https://docs.docker.com/go/attack-surface/
$ echo $CI_JOB_TOKEN | docker login -u $CI_REGISTRY_USER --password-stdin $CI_REGISTRY
Error response from daemon: Get "https://registry.gitlab.com/v2/": dial tcp: lookup registry.gitlab.com on 192.168.0.1:53: no such host
Cleaning up project directory and file based variables 00:00
ERROR: Job failed: exit code 1

Versions

  • Self-managed
  • GitLab.com SaaS
  • Self-hosted Runners

Versions

  • GitLab (Web: /help or self-managed system information): Enterprise Edition 17.0.0-pre f78429a31a9
  • GitLab Runner, self-hosted gitlab-runner --version: 16.11.0
    :warning: I’m running the gitlab-runner in a docker container (privileged mode as well, which did not make any difference)

Any help would be greatly appreciated!