Docker-in-Docker i/o timeout

GitLab CI/CD
1

Problem to solve

When I try to run the CI/CD job below in GitLab I get the following error:

#1 DONE 0.0s
#2 [internal] load metadata for registry.gitlab.com/islandoftex/images/texlive:latest
#2 ERROR: failed to do request: Head "https://registry.gitlab.com/v2/islandoftex/images/texlive/manifests/latest": dial tcp 35.227.35.254:443: i/o timeout
------
 > [internal] load metadata for registry.gitlab.com/islandoftex/images/texlive:latest:
------
Dockerfile:1
--------------------
   1 | >>> FROM registry.gitlab.com/islandoftex/images/texlive:latest
   2 |     
   3 |     SHELL ["/bin/bash", "-c"]
--------------------
ERROR: failed to build: failed to solve: DeadlineExceeded: registry.gitlab.com/islandoftex/images/texlive:latest: failed to resolve source metadata for registry.gitlab.com/islandoftex/images/texlive:latest: failed to do request: Head "https://registry.gitlab.com/v2/islandoftex/images/texlive/manifests/latest": dial tcp 35.227.35.254:443: i/o timeout

Steps to reproduce

I created a privileged GitLab runner (Docker executor) and I have tried to execute the Docker-in-Dolcer CI/CD job below.

Configuration

variables:
  DOCKER_HOST: tcp://docker:2376
  DOCKER_TLS_CERTDIR: "/certs"
  DOCKER_TLS_VERIFY: "1"
  DOCKER_CERT_PATH: "$DOCKER_TLS_CERTDIR/client"

default:
  image: docker:29
  services:
    - name: docker:29-dind
      alias: docker
  before_script:
    - docker info
  tags:
    - dind

build:
  stage: build
  rules:
    - if: $CI_PIPELINE_SOURCE == "schedule"
  script:
    - export VERSION="`date +%F`"
    - docker login -u gitlab-ci-token -p $CI_JOB_TOKEN $CI_REGISTRY
    - docker build -t $CI_REGISTRY_IMAGE:$VERSION -t $CI_REGISTRY_IMAGE:latest --build-arg VERSION=ECC-$VERSION --build-arg NAME=ECC-$VERSION .
    - docker push $CI_REGISTRY_IMAGE:latest
    - docker push $CI_REGISTRY_IMAGE:$VERSION
  artifacts:
    expire_in: "1 hr"

Versions

Please select whether options apply, and add the version information.

  • Self-managed
  • GitLab.com SaaS
  • Dedicated
  • Self-hosted Runners

Versions

System information
System:		
Current User:	git
Using RVM:	no
Ruby Version:	3.3.12
Gem Version:	3.7.2
Bundler Version:4.0.18
Rake Version:	13.4.2
Redis Version:	7.2.16
Sidekiq Version:7.3.9
Go Version:	unknown

GitLab information
Version:	19.5.1
Revision:	449ed2875c2
Directory:	/opt/gitlab/embedded/service/gitlab-rails
DB Adapter:	PostgreSQL
DB Version:	17.11
URL:		https://git.example.dk
HTTP Clone URL:	https://git.example.dk/some-group/some-project.git
SSH Clone URL:	git@git.example.dk:some-group/some-project.git
Using LDAP:	no
Using Omniauth:	yes
Omniauth Providers: 

GitLab Shell
Version:	14.58.0
Repository storages:
- default: 	unix:/var/opt/gitlab/gitaly/gitaly.socket
GitLab Shell path:		/opt/gitlab/embedded/service/gitlab-shell

Gitaly
- default Address: 	unix:/var/opt/gitlab/gitaly/gitaly.socket
- default Version: 	19.5.1-g0340305
- default Git Version: 	2.55.0-rc1.g02bb39c

# gitlab-runner --version
Version:      19.5.1
Git revision: f411f264
Git branch:   19-5-stable
GO version:   go1.26.10
Built:        2026-10-22T12:38:01Z
OS/Arch:      linux/amd64
2

Hi @madsmh,

thanks for sharing the full config and the error output. That makes it much easier to dig in.

The key part is dial tcp 35.227.35.254:443: i/o timeout. It means the TCP connection to registry.gitlab.com never got established, so this is not the registry being slow. The request never reaches it. docker info works, so the dind service is fine. The traffic from your runner network out to the internet is getting stuck somewhere.

Worth knowing: docker login against $CI_REGISTRY works because that is your self-managed instance on the internal network. It doesn’t show whether you can reach external registries.

Let’s narrow it down with three checks:

  1. On the runner host itself:

    curl -sv https://registry.gitlab.com/v2/ -o /dev/null

  2. In the job container, as the first line in script:

    wget -qO- --timeout=10 https://registry.gitlab.com/v2/ ; echo $?

  3. Inside the dind network, which is where the FROM pull happens:

    docker run --rm curlimages/curl -sv --max-time 10 https://registry.gitlab.com/v2/

How to read the results:

  • The host fails too: a firewall or network policy is blocking outbound 443. Many company networks require an HTTP proxy. In that case, set HTTP_PROXY, HTTPS_PROXY and NO_PROXY as variables on the docker:29-dind service as well, not only on the job, because the dind daemon does the image pull. Another option is the Dependency Proxy or a pull-through cache on your instance, so the builds don’t need direct internet access.
  • The host works, containers fail: look at host firewall rules (firewalld, ufw, custom iptables/nftables) that might drop forwarded traffic from the Docker bridge networks. If the host runs on OpenStack, a VPN or an overlay network, also compare the MTU from ip link with the dind default of 1500. An MTU mismatch usually looks more like a TLS hang, but it’s cheap to rule out.

One more question: did this job ever work on this runner, or is it a new setup? If it worked before, a recent network or host change is the first thing to look at.

Let us know what the three checks show.

3

I am using an OVH VPS, the first check passed, the second, however did not, I got,

$ wget -qO- --timeout=10 https://registry.gitlab.com/v2/ ; echo $?
wget: download timed out

From the GitLab CICD output

4

The host can reach registry.gitlab.com and the job container can’t. The job container doesn’t send its traffic through dind, so the problem sits below dind: the VPS drops container traffic, or doesn’t NAT it, before it leaves the host. It’s a timeout, not a lookup failure, so DNS works.

Please run these on the VPS as root:

1. Which network do jobs use?

grep -nE 'network_mode|FF_NETWORK_PER_BUILD' /etc/gitlab-runner/config.toml

No output means jobs run on Docker’s default bridge network.

2. Bridge network vs. host network

docker run --rm alpine wget -qO- --timeout=10 https://registry.gitlab.com/v2/
docker run --rm --network host alpine wget -qO- --timeout=10 https://registry.gitlab.com/v2/

HTTP/1.1 401 Unauthorized is the good result: the registry answered and asked for authentication. download timed out means the traffic is blocked. If the first command times out and the second returns 401, Docker’s forwarding or NAT on the host is broken.

3. Forwarding and Docker’s firewall rules

grep PRETTY_NAME /etc/os-release
sysctl net.ipv4.ip_forward
nft list ruleset | grep -iE 'docker|masquerade'

You should see ip_forward = 1, Docker chains, and a masquerade rule for the Docker subnet (usually 172.17.0.0/16). That rule rewrites the container’s source IP to the VPS’s public IP. Without it, the registry’s replies never come back.

4. Host firewall

ufw status verbose
systemctl is-active firewalld

The firewall treats container traffic as forwarded traffic, not as the host’s own outgoing traffic. That’s why the host works and containers don’t.

Likely cause: a firewall reload or a custom nftables script removed the rules Docker created at startup. systemctl restart docker creates them again. Heads-up: the restart also restarts all containers, including the runner and GitLab if they run in Docker. If it fixes the problem, make sure the firewall starts before Docker and doesn’t flush Docker’s rules on reload.

5

Sorry, I made a silly mistake! I ran test no. on the wrong VPS, the one that GitLab has been migrated to, fails Test 1 as well:

Host registry.gitlab.com:443 was resolved.

IPv6: (none)

IPv4: 35.227.35.254

Trying 35.227.35.254:443...

connect to 35.227.35.254 port 443 from 51.83.134.31 port 55006 failed: Connection timed out

Failed to connect to registry.gitlab.com port 443 after 135027 ms: Could not connect to server

closing connection #0