Project won't pick up docker executor runners

How to Use GitLab
1

I’ve registered a couple runners with the intent of building docker images with these. I’ve tried several variations, including using the shell executor as well as docker-in-docker and docker socket binding but nothing seems to work.

For instance, after installing docker + gitlab-runner on a VM, I’ve registered a specific runner on a project using the following:
gitlab-runner register -n --url https://gitlab.example.com/ --registration-token xxxxx --executor docker --description "docker runner" --docker-image "docker:stable" --docker-volumes /var/run/docker.sock:/var/run/docker.sock

I can see that this runner is available in the project under CI/CD runners, and I can see that it is active. But if I try to run the CI/CD pipeline, it never gets picked up and sits at `This job is in pending state and is waiting to be picked by a runner"

The version of the runner im using is 12.5.0, and the version of gitlab is 11.9.7.

If I containerize the runner, i get a little more success - so by running:

docker run -d --name gitlab-runner1 --restart always   -v /app/git/config/runner1:/etc/gitlab-runner   -v /var/run/docker.sock:/var/run/docker.sock   gitlab/gitlab-runner:latest

and then registering it:

root@a23b8260b50f:/# gitlab-runner register -n --url https://gitlab.example.com/ --registration-token xxxxxx --executor docker --description "docker runner q9b - test" --docker-image "docker:stable" --docker-volumes 
/var/run/docker.sock:/var/run/docker.sock

root@a23b8260b50f:/# cat /etc/gitlab-runner/config.toml
concurrent = 1
check_interval = 0

[session_server]
  session_timeout = 1800

[[runners]]
  name = "docker runner"
  url = "https://gitlab.example.com/"
  token = "xxxxxx"
  executor = "docker"
  [runners.docker]
    tls_verify = false
    image = "docker:stable"
    privileged = false
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = false
    volumes = ["/var/run/docker.sock:/var/run/docker.sock", "/cache"]
    shm_size = 0
  [runners.cache]
    [runners.cache.s3]
    [runners.cache.gcs]

but than the jobs fail with the following error:
Cannot connect to the Docker daemon at tcp://localhost:2375. Is the docker daemon running?

If anyone could help make sense out of this it would be really appreciated.

2

Couple of clarifications since you don’t provide a gitlab-ci.yml file:

  1. Did you set the runner to be able to take up unregistered jobs? If not go to your runner settings inside of gitlab and uncheck the box for this (runners section, then edit). Alternatively register the runner with tags and put the tags into your gitlab-ci.
  2. Are you running another job parallel? The concurrent = 1 value allows only for one job to run simultaneously.

If you checked the above:

I am not sure if that works but if you use docker in docker you can try:

  1. Set privileged to true (you have to do this when using dind)
  2. Put this in your gitlab-ci:
image: docker:19.03.0 ##or whatever docker image you use

variables:
  DOCKER_DRIVER: overlay2
  # Create the certificates inside this directory for both the server
  # and client. The certificates used by the client will be created in
  # /certs/client so we only need to share this directory with the
  # volume mount in `config.toml`.
  DOCKER_TLS_CERTDIR: "/certs"

services:
  - docker:19.03.0-dind
  1. Finally in your config.toml:
    volumes = ["/certs/client", “/cache”]

Hope this helps

3

Did you set the runner to be able to take up unregistered jobs? If not go to your runner settings inside of gitlab and uncheck the box for this (runners section, then edit). Alternatively register the runner with tags and put the tags into your gitlab-ci.

Yep, i made sure the runner can pick up jobs without tags in the runner settings.

Are you running another job parallel? The concurrent = 1 value allows only for one job to run simultaneously.

No, this is a brand new environment, nothing else is running here and this is the first job im trying to get started.

Set privileged to true (you have to do this when using dind)

For now, let’s assume that i’m simply trying to use gitlab-runner installed on bare-metal and i’m trying to leverage the Docker executor.

  • After registering a specific runner to my project, I can see that the runner is present in the project.
  • I run a quick gitlab-runner verify on the runner to make sure everything is working fine, and the runner is showing as green.

I didn’t provide the gitlab-ci.yaml because the job isn’t getting picked up, and although I included a tidbit about it in my previous post, I added that purely to demonstrate that the servers can speak to one another - Docker in docker is not the solution i’m opting for at this time.

So, when I’m trying to determine is why, using a fresh install of Gitlab-Runner on VM, will the Job not get picked up?

The command i used to register the runner:

gitlab-runner register -n --url https://git.example.com/ --registration-token xxxxxxxxxxxx --executor docker --description "docker runner " --docker-image "docker:stable" --docker-volumes /var/run/docker.sock:/var/run/docker.sock

And the gitlab-ci.yaml is only running a docker info command:

stages:
    - build

Build image:
    stage: build
    script:
        - docker info
4

Hmm ok very odd indeed. Ok I don’t know why it doesn’t work for you but here is some ideas what you could try (no guarantee whatsoever):

  1. you can run your runner in debug mode gitlab-runner --debug <command>
  2. you can try and update your gitlab version. 11.9.7. is old.
  3. Register your runner without the terminal command but just type sudo gitlab-runner register and then follow the instructions
  4. Register your runner with tags and the tag your job in your gitlab-ci
  5. I am pretty sure that the error Cannot connect to the Docker daemon at tcp://localhost:2375. Is the docker daemon running? is occurring when using dind and you can get rid of it by my suggestions above

Maybe doing this will give you some clues where things go wrong…

Also, are you running your runner inside a docker container?

5

Thanks for your prompt response - something that did just dawn on me (and your reply confirms that it will be worth testing), is that I’m not registering the runners using sudo. I’ll try running in debug as you suggest, I hope this help. I also tried upgraded gitlab to 12.5 and still have issues unfortunately.

Tagging might also be a good idea.

I am pretty sure that the error Cannot connect to the Docker daemon at tcp://localhost:2375. Is the docker daemon running? is occurring when using dind and you can get rid of it by my suggestions above

If all else fails, I’ll try giving DIND a shot and i’ll look at using your suggestion.

Thanks again!

6

Ok sounds like you have some options getting this to work. Using sudo might definitely help. And while you’re at it, maybe also updating the runner could help. Idk what the newest version is though or if u already using it. Good luck and let me know if it worked out or into what errors you run on your way…

7

It seems sudo was the problem, the jobs are getting picked up now. I’ll go ahead and re-run my previous tests, that’s really a bit surprising

8

fantastic! Happy you got it working.