Windows docker runner error "invalid IP address in add-host"

I have a Windows EC2 instance in AWS, with Docker installed. I have two GitLab runners configured on this instance: a PowerShell runner (which runs jobs on the VM itself) and a Docker runner (that should run jobs using a container image specified in the .gitlab-ci.yml

I have a Windows container image, based on mcr.microsoft.com/dotnet/framework/sdk:4.7.2-20191008-windowsservercore-ltsc2019, with some additional tools installed.

When I try to build a simple console hello-world program using the container, I get this message as soon as the build starts:

ERROR: Job failed (system failure): prepare environment: Error response from daemon: invalid IP address in add-host: “172.27.84.100 172.27.82.245” (docker.go:651:0s). Check Types of shells supported by GitLab Runner | GitLab for more information

The two IP addresses listed in the error message do not match the IP address of the instance or its default gateway.

Based on the link in the error message, I verified that the OS version on the EC2 instance matches the OS version inside the container.

OS Name: Microsoft Windows Server 2019 Datacenter
OS Version: 10.0.17763 N/A Build 17763

I get the same error if I run a minimal .gitlab-ci.yml file that references a Microsoft-built container ( mcr.microsoft.com/windows/servercore:1809) and simply tries to execute a PowerShell command. This suggests that the problem is with Docker, rather than with the containers. The “add-host” in the error message suggests the “–add-host” option to the “docker run” command, which is supposed to “add a custom host-to-ip mapping” (per the Docker CLI documentation).

I can run containers manually from the command line on the instance (including the containers noted above); those containers start and behave normally.

As a test, I updated Docker from 19.03 (our standard version) to 20.10.7; this did not change the behavior.

Both the GitLab runner and the Docker service are running as users with Administrator rights on the instance.

I updated config.toml with log_level = “debug” and restarted the services, but I do not see any additional information in Event Viewer. The error happens so early in the process (failing to start the container) that there is no more information to be gained by increasing the log level.

One possibility I have not thoroughly explored: this EC2 instance was launched from an AMI built using Packer, with Docker installed during the Packer build process. It is possible that one of the IP addresses is one that the source instance had at the time that Packer was installing Docker on it. But if that were the case, I would expect to see at least one of the IP addresses in the error message be the same on different instances, and what I see is that both IP addresses change when I move to a new instance.

Here is the version information for gitlab-runner.exe:
Version: 14.7.0
Git revision: 98daeee0
Git branch: 14-7-stable
GO version: go1.17.5
Built: 2022-01-19T17:11:50+0000
OS/Arch: windows/amd64

Any advice would be greatly appreciated. Thanks in advance.

This issue was resolved with help from GitLab support–many thanks to Elif Munn for her help. I’m posting the resolution and root cause in case it helps someone else in the future.

The issue was caused by a bad line in the config.toml file for the Docker runner. The line looked like this:

extra_hosts = [“gitlab.toolchain.corning.com:172.27.82.120 172.27.85.204”]

A correctly formatted line would look like this:

extra_hosts = [“gitlab.toolchain.corning.com:172.27.84.100”, “gitlab.toolchain.corning.com:172.27.82.245”]

This line ended up in the config.toml file because the PowerShell script that installs the GitLab runner does a DNS lookup on our GitLab server in order to construct that line. We have a load-balanced instance, so the lookup returns two IP addresses, separated by a space. The script makes the (usually reasonable) assumption that only one IP address will be returned, and constructs the line accordingly.

While it is possible to update the PowerShell script to construct the extra_hosts line correctly, we discovered that the line is not needed in the config.toml, so that parameter was removed.