Run runners in parallel on the same server

GitLab CI/CD
1

I have four runners at the same server, let’s call them runner[1234].
I now have a project which lints at three php versions. These builds do not depend on each other, and I’ve told GitLab CI to run them in parallel. In fact, they don’t do that. Each runner is shared, the project is allowed to use all of them. So the situation was: runner2 picks php-lint for 5.4, then, after it finished, runner4 picks php-lint for 5.6, and then after that finished, runner3 picks php-lint for 7.0.

To avoid this, I’ve set concurrent = 4 at config.toml 4 is the number of runners, so my intention was, that each runner picks one job, and proceeds it. Instead, runner2 picked all three php-lints and executed them at the same time. (All runners are located at the same server).

Then I’ve changed my config to something like this:
concurrent = 4
check_interval = 0

[[runners]]
name = “runner-1”
url = “…”
token = “…”
executor = “docker”
concurrent = 1
[runners.docker]

So I’ve tried to tell the runners: Run up to 4 jobs, but only one per runner. That failed, runner2 claimed still all jobs. Is there a way to reach what I’ve tried? Each runner can pick one job at the same time?

2

I’ve found the solution.

For people running into the same problem: Set concurrent = number of your runners, and add the “limit” param to each docker section. There you can set limit = 1, and then each docker will pick one job add the same time.

(Topic is resolved, but I don’t found a way to close it myself).

1 Like
3

Do you happen to have the full config.toml (excerpt) for this available? I tried setting up multiple runners.docker1-2-3-4 blocks but this doesn’t really seem to be the way…

Edit: I think I managed to figure it out. For reference, something like this:

[[runners]]
  name = "gitlab-ci-1"
  url = "https://foo-matic.acme.inc/"
  token = "top-secret-token"
  executor = "docker"

  [runners.docker]
    tls_verify = false
    image = "openjdk:11-jdk-slim-sid"
    privileged = false
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = false
    volumes = ["/cache"]
    shm_size = 0
    limit = 1

[[runners]]
  name = "gitlab-ci-2"
  url = "https://foo-matic.acme.inc/"
  token = "top-secret-token"
  executor = "docker"

  [runners.docker]
    tls_verify = false
    image = "openjdk:11-jdk-slim-sid"
    privileged = false
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = false
    volumes = ["/cache"]
    shm_size = 0
    limit = 1

More details: https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-runners-section

4

Thanks for the linkto the docs. In my case, I just registered one runner but wanted it to run several jobs at once. This is my configuration file, most of it is auto generated.

concurrent = 8
check_interval = 10

[session_server]
  session_timeout = 1800

[[runners]]
  name = "raspberrypi"
  url = "https://gitlab.com"
  token = "****************************"
  executor = "docker"
  [runners.custom_build_dir]
  [runners.cache]
    [runners.cache.s3]
    [runners.cache.gcs]
    [runners.cache.azure]
  [runners.docker]
    tls_verify = false
    image = "python3"
    privileged = false
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = false
    volumes = ["/cache"]
    shm_size = 0

Note:

  • concurrent = 8 is set at the top, allowing up to 8 jobs in parallel.
  • limit = 0 is the default in the runner called raspberrypi which means that it can take as many jobs as it wants.

Result: The runner raspberrypi can run up to 8 jobs in parallel.