Gitlab Runner Within Docker Container Cannot Communicate With Gitlab Repository

How to Use GitLab
,
1

I’m creating a demo Gitlab environment using two AWS EC2 instances, one for the CI and one for the runner. They’re both using the Ubuntu 18.04 LTS image and both have public/private IPv4 addresses. The security groups between the two are pretty open, all TCP within the AWS /28 network both EC2 instances sit in, from our office and from our VPN devices are allowed inbound on both machines.

I’ve followed this documentation that is a bit dated, but goes through the steps of registering a Docker container on the runner that utilizes a Docker-in-Docker (DIND) image to spawn containers within itself to run our builds. I was able to get through the setup and configuration, the runner is installed and registered to the CI. However, whenever a checkout / pull occurs at the beginning of the job, the git procedure times out unable to reach the CI’s repo. Additionally, taking the string and using the runner’s token I cannot clone the repo when I run the command locally, it complains “HTTP basic: Access Denied.”

I’m a bit stuck at this point and don’t really know how to get the runner to be able to properly checkout / pull our Gitlab repo code into the job workspace. Testing this when the runner was on the EC2 instance instead of being inside a Docker container did not have this problem. Here are the configuration steps I took, some PII was changed.

Domain used: welovepets.com
Runner Config:

  • apt-get install apt-transport-https ca-certificates curl gnupg-agent software-properties-common

Install Docker:

  • curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
  • add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu bionic stable"
  • apt-get update && apt-get install -yqq docker-ce docker-ce-cli containerd.io

Runner Post-steps:

  • groupadd docker (should already exists) && usermod -aG docker ubuntu
  • systemctl enable docker, restart services

Install Gitlab-Runner:

  • curl -L https://packages.gitlab.com/install/repositories/runner/gitlab-runner/script.deb.sh | sudo bash
  • sudo apt-get install gitlab-runner

Gitlab-Runner Post-steps:

  • Confirm overlay with lsmod | grep overlay

  • Create /etc/docker/daemon.json with { "storage-driver": "overlay2" }

  • systemctl restart docker

  • docker network create gitlab-runner-network

  • Create DIND container:

    • docker run -d --name gitlab-runner-dind --privileged --restart always --network gitlab-runner-network -v /var/lib/docker docker:18.06.3-ce-dind --storage-driver=overlay2
    • mkdir -p /srv/gitlab-runner && touch /srv/gitlab-runner/config.toml

  • Create Runner within DIND container:

    • docker run -d --name gitlab-runner --restart always --network gitlab-runner-network -v /srv/gitlab-runner/config.toml:/etc/gitlab-runner/config.toml -e DOCKER_HOST=tcp://gitlab-runner-dind:2375 gitlab/gitlab-runner:ubuntu

  • Register the runner:

    • docker run -it --rm -v /srv/gitlab-runner/config.toml:/etc/gitlab-runner/config.toml gitlab/gitlab-runner:ubuntu register --tag-list "docker,runner" --registration-token "<TOKEN>" --executor docker --run-untagged=true --locked=false --description "Docker DIND Runner" --url "http://gitlab-demo.welovepets.com/" --docker-image docker:18.06.3-ce --docker-volumes /var/run/docker.sock:/var/run/docker.sock
  • Create Gitlab group “dogs”, add my user account to it
  • Create Gitlab project “bones” associated with group, my user account is automatically made Owner
  • Push changes successfully using my user account credentials & Personal Access Token
  • Pipeline engages, job starts within it’s own container on the gitlab-runner host

Gitlab-Runner config.toml:

check_interval = 0

[session_server]
  session_timeout = 1800

[[runners]]
  name = "Docker DIND Runner"
  url = "http://gitlab-demo.welovepets.com/"
  token = "<TOKEN>"
  executor = "docker"
  [runners.custom_build_dir]
  [runners.docker]
    extra_hosts = ["[gitlab-demo.welovepets.com]:[<GITLAB_CI_PRIVATE_IPV4_ADDRESS>]"]
    tls_verify = false
    image = "docker:18.06.3-ce"
    privileged = false
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = false
    volumes = ["/var/run/docker.sock:/var/run/docker.sock", "/cache"]
    shm_size = 0
  [runners.cache]
    [runners.cache.s3]
    [runners.cache.gcs]

Results:

  • Job fails after a 2 minute “Operation timed out” error while fetching the repo changes:
  on Docker DIND Runner 6YA9iN-6
Using Docker executor with image docker:18.06.3-ce ...
Pulling docker image docker:18.06.3-ce ...
Using docker image sha256:163a8770288acb45fbce8d83bd7dd569731302a992d0cc6eec076834b14543fd for docker:18.06.3-ce ...
Running on runner-6YA9iN-6-project-1-concurrent-0 via ddfc52d20579...
Initialized empty Git repository in /builds/dogs/bones/.git/
Fetching changes...
Created fresh repository.
fatal: unable to access 'http://gitlab-ci-token:[MASKED]@gitlab-demo.welovepets.com/dogs/bones.git/': Failed to connect to gitlab-demo.welovepets.com port 80: Operation timed out
ERROR: Job failed: exit code 1
  • Attempting manual clone using runner’s token also fails:
Cloning into 'bones'...
remote: HTTP Basic: Access denied
fatal: Authentication failed for 'http://gitlab-ci-token:<TOKEN>@gitlab-demo.welovepets.com/dogs/bones.git/'
  • Same repo, using my own user credentials & personal access token works fine:
Cloning into 'bones'...
Username for 'http://gitlab-demo.welovepets.com': nraymond
Password for 'http://nraymond@gitlab-demo.welovepets.com':
remote: Enumerating objects: 28, done.
remote: Counting objects: 100% (28/28), done.
remote: Compressing objects: 100% (12/12), done.
remote: Total 505 (delta 16), reused 28 (delta 16)
Receiving objects: 100% (505/505), 496.70 KiB | 7.20 MiB/s, done.
Resolving deltas: 100% (183/183), done.

Any ideas as to why the spawned gitlab-runner container inside the DIND container on the Gitlab-Runner EC2 instance cannot communicate with the repository on the Gitlab CI EC2 instance would be much appreciated, thanks!