GitLab-Runner is unable to resolve custom domain

Hi,

i’m currently trying to run a simple CI job, that runs a curl inside a curl container. Unfortunately, somehow, the domain, that is called by curl seems not to be resolvable by the container.

This is the log output of my runner:

Using docker image sha256:e5da2657681780567a06d03fa480dda5b16cdd91bfbe269fe9a1d9adcdc0fc4d for curlimages/curl:latest with digest curlimages/curl@sha256:4a3396ae573c44932d06ba33f8696db4429c419da87cbdc82965ee96a37dd0af ...
$ curl --location --request POST "http://mydomain.de/autoupload?secret=${IMPORTER_SECRET}" \ # collapsed multi-line command
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (6) Could not resolve host: ffi.fuchscloud.de
Cleaning up project directory and file based variables
00:00
ERROR: Job failed: exit code 6

My .gitlab-ci.yml

stages:
  - cron
image: curlimages/curl:latest

run-cron:
  stage: cron
  tags:
    - docker
  script:
    - |
     curl --location --request POST "http://mydomain.de/autoupload?secret=${IMPORTER_SECRET}" \
     --header "Accept: application/json" \
     --header "Authorization: Bearer ${TOKEN}" \
     --form 'json=@"./config.json"'

I can ping mydomain.de from the shell of the gitlab runner. I can also ping mydomain.de from inside the curl docker container

docker run -it curlimages/curl:latest ping mydomain.de
PING mydomain.de (192.168.2.200): 56 data bytes
64 bytes from 192.168.2.200: seq=0 ttl=42 time=0.316 ms
64 bytes from 192.168.2.200: seq=1 ttl=42 time=0.236 ms

BUT, sometimes the ping does not work from inside the container. While testing it for this thread, the first time, it did not work.

docker run -it curlimages/curl:latest ping mydomain.de
ping: bad address 'mydomain.de'

I have the feeling, that this effect takes place when i run my scheduled job.

I have the following architecture:
image

I’m using a Raspberry-PI with pihole as my DNS. I have setup the custom domain mydomain.com within pihole. I can ping and access the web-frontend of mydomain.com. GitLab is running inside a LXC container and the GitLab runner is a Ubuntu 22.04 LTS VM.

After some googling i found a fix, that i should just add my custom DNS in my runner’s config.toml. This is what my config.toml looks like now (192.168.2.6 is the IP of my Raspberry-PI). But after applying this change, the runner still has issues to resolve the domain.

concurrent = 1
check_interval = 0
shutdown_timeout = 0

[session_server]
  session_timeout = 1800

[[runners]]
  name = "vm-ubuntu-gitlab-runner"
  url = "https://git.mydomain.de"
  id = 10
  token = "redacted"
  token_obtained_at = 2023-08-22T06:03:09Z
  token_expires_at = 0001-01-01T00:00:00Z
  executor = "docker"
  dns = [ "192.168.2.6", "1.1.1.1"  ]
  [runners.docker]
    tls_verify = false
    image = "docker"
    privileged = false
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = false
    volumes = ["/cache", "/var/run/docker.sock:/var/run/docker.sock"]
    shm_size = 0

Any Idea, what could be wrong?

Apperently, removing the cloudflare dns 1.1.1.1 from my config.toml solved the issue for me.

I would assume depending on what DNS resolver resolved first is the issue. Your pihole probably gives a nice “192.168.2.10” address or something whereas cloudflare would, if any is set externally at all, respond with a 82.95.100.105 address and refuse to curl.

As they say- it can’t be DNS, it isn’t DNS, it was DNS.

Yes, this is because you had two DNS entries configured. One gave the correct replies, the second (Cloudflare) didn’t have the entries configured or wrong IP’s and thus it wouldn’t work. DNS is cyclic for queries, so it will ask alternatively one or the other server. So you could more or less have a 50% success rate in resolving it. As soon as you removed the Cloudflare one, you were only using the piHole and then got 100% response rate for DNS queries.

If using multiple DNS servers, they must both resolve the same IP’s, and also have all the DNS entries required. Any missing entry on either of the DNS servers would cause an issue.

You can actually configure entries in Cloudflare DNS to resolve just to local IP addresses. In which case, had you replicated those entries that were being resolved by your piHole, you could then still use the Cloudflare DNS entry. I wouldn’t personally recommend doing that, but it is possible. Cloudflare doesn’t block DNS entries from being created with private/local IP’s.

Hi thanks for the reply.

i guess i’m fine with my current solution. I wouldn’t need to configure a “global” DNS, just to have it working in my local network :-). That’s actually the reason i am using my pihole.
I wasn’t sure about the second DNS. I know, that a lot of devices allow to configure multiple DNS, just in case, the primary does not respond. But i guess in my case i have to rely on my local DNS.