Gitlab runner ssh executer takes a long time to pick up jobs

GitLab CI/CD
1

Hi,

I am using a selfhosted gitlab runner, with 4 registered runners.
1 is a docker executor and 3 are ssh runners for different servers. The docker executor is running fine, but the ssh runners need a very long time to pick up jobs, which leads to long queue times althugh there are no other jobs running and there is no limit on how much jobs can be run in parallel.

Screenshot of jobs of the first ssh runner:

image
image1489×493 44.5 KB

Here you can see, that two jobs, although nothing was running for a longer time, got a queue time of more then 20 minutes.

gitlab-runner.toml

check_interval = 0
user = "gitlab-runner"
shutdown_timeout = 0

[[runners]]
  name = "runner 1"
  url = "https://gitlab.com"
  id = <removed>
  token = "<removed>"
  token_obtained_at =<removed>
  token_expires_at = <removed>
  executor = "docker"
  shell = "sh"
  limit = 4
  [runners.cache]
    Type = "s3"
    Shared = true
    MaxUploadedArchiveSize = 0
    [runners.cache.s3]
      ServerAddress = "<removed>"
      AccessKey = "<removed>"
      SecretKey = "<removed>"
      BucketName = "runners-cache"
  [runners.docker]
    tls_verify = false
    image = "ubuntu:latest"
    memory = "8g"
    cpus = "2"
    privileged = false
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = false
    volumes = ["/cache"]
    shm_size = 0
    network_mtu = 0

[[runners]]
  name = "<removed>"
  url = "https://gitlab.com/"
  id = <removed>
  token = "<removed>"
  token_obtained_at = <removed>
  token_expires_at = <removed>
  executor = "ssh"
  [runners.ssh]
    user = "<removed>"
    host = "<removed>"
    port = "<removed>"
    password = ""
    identity_file = "<removed>"
    disable_strict_host_key_checking = true

[[runners]]
  name = "<removed>"
  url = "https://gitlab.com"
  id = <removed>
  token = "<removed>"
  token_obtained_at = <removed>
  token_expires_at = <removed>
  executor = "ssh"
  [runners.ssh]
    user = "<removed>"
    host = "<removed>"
    port = "22"
    password = ""
    identity_file = "<removed>"
    disable_strict_host_key_checking = true

[[runners]]
  name = "<removed>"
  url = "https://gitlab.com"
  id = <removed>
  token = "<removed>"
  token_obtained_at = <removed>
  token_expires_at = <removed>
  executor = "ssh"
  [runners.ssh]
    user = "<removed>"
    host = "<removed>"
    port = "<removed>"
    password = ""
    identity_file = "<removed>"
    disable_strict_host_key_checking = true

The toml is gettting accepted and there is no error accept for this warning: [SSH Executor] error in config.toml file (#31105) · Issues · GitLab.org / gitlab-runner · GitLab

Can anyone help me to reduce these queue times to a normal 10 seconds or help me to find a way to debug this?

2

If someone else comes across this issue:
The error was the missing concurrent = <number> flag in the global section. (See Advanced configuration | GitLab)
If not set it seems like it defaults to 1.