Parallel:matrix isn't executing in parallel

I am trying to get a job to run in parallel within gitlab. I have registered 8 runners on one server and set concurrent=16 in /etc/gitlab-runner/config.toml. I see all 8 runners configured within that file. Below is the beginning of the file:

concurrent = 16
check_interval = 0

[session_server]
  session_timeout = 1800

[[runners]]
  name = "gitlab-runner-02-#1"
  url = "http://gitlab.example.com"
  token = "redacted"
  executor = "docker"
  [runners.custom_build_dir]
  [runners.cache]
    [runners.cache.s3]
    [runners.cache.gcs]
    [runners.cache.azure]
  [runners.docker]
    tls_verify = false
    image = "almalinux:latest"
    privileged = false
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = false
    volumes = ["/cache"]
    shm_size = 0

[[runners]]
  name = "gitlab-runner-02-#2"
  url = "http://gitlab.example.com"
  token = "redacted"
  executor = "docker"
  [runners.custom_build_dir]
  [runners.cache]
    [runners.cache.s3]
    [runners.cache.gcs]
    [runners.cache.azure]
  [runners.docker]
    tls_verify = false
    image = "almalinux:latest"
    privileged = false
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = false
    volumes = ["/cache"]
    shm_size = 0

My gitlab-ci.yml:

image: almalinux:latest

stages:
  - build

foo:
  stage: build
  script: sleep 60
  parallel:
    matrix:
      - foo: [a, b, c, d, e, f, g, h]

Rather than executing each job independently on each of my 8 runners, it executes a single job at a time. From everything I’ve read, this should be enough for it to run in parallel. I’ve spent over an hour trying to find someone else who has run across the same problem I have. From what I have read, usually other people have forgotten to set the “concurrent” option in their config, but that is not the case here.

The gitlab instance in question is self managed and on the free tier.

I would greatly appreciate any help with this issue! For full transparency, I have crossposted on stack overflow (gitlab parallel:matrix does not run concurrently - Stack Overflow). I will make sure both stackoverflow and this forum have the answer clearly described once I find it.

Forgot to mention, we are on version v14.10.0-ee.

I seem to have resolved the issue. I need to do more experimenting, but it appears to be related to another registered runner on a different server not allowing more than one job. I also increased the limit for each individual runner. I need to do more experimenting to understand what actually fixed it still.

Hi,

We have a similar problem, did you find why ?

I’m sorry, I don’t remember what exactly fixed it.