[SOLVED] How to figure out why all my runners aren't being used?

I’ve got 4 instances of docker runners configured on a single machine I spun up on AWS.
For some reason there seems to only be 1 jobs being run when I look at the current jobs on gitlab.com (this is confirmed with the prometheus /metrics endpoint on the gitlab runner showing only one runner job at the moment).
However, there are 3 pending jobs with the same tags!
And i’ve observed that once the single job finishes, one of the others gets picked up, but only one at a time!

I have no idea why it’s not picking up 4 jobs to run at a time, given there are 4 available runners.

I’ve run gitlab-runner verify and also checked the runner configuration on gitlab.com and via API, all of them say that all runners are active and online, ready to accept jobs:

[
  {
        "id": [REDACTED],
        "description": "AWS-manual-docker-runner",
        "ip_address": [REDACTED],
        "active": true,
        "is_shared": false,
        "runner_type": "group_type",
        "name": "gitlab-runner",
        "online": true,
        "status": "online"
    },
    {
        "id": [REDACTED],
        "description": "AWS-manual-docker-runner-2",
        "ip_address": [REDACTED],
        "active": true,
        "is_shared": false,
        "runner_type": "group_type",
        "name": "gitlab-runner",
        "online": true,
        "status": "online"
    },
    {
        "id": [REDACTED],
        "description": "AWS-manual-docker-runner-3",
        "ip_address": [REDACTED],
        "active": true,
        "is_shared": false,
        "runner_type": "group_type",
        "name": "gitlab-runner",
        "online": true,
        "status": "online"
    },
    {
        "id": [REDACTED],
        "description": "AWS-manual-docker-runner-4",
        "ip_address": [REDACTED],
        "active": true,
        "is_shared": false,
        "runner_type": "group_type",
        "name": "gitlab-runner",
        "online": true,
        "status": "online"
    }
]

A bit of system information:
The runners are configured on an AWS EC2 instance with gitlab-runner installed. There are 4 instances of docker runners configured on the machine, all with the same tags and configuration:

image428×986 27.8 KB

EDIT:
Still seeing multple pending jobs with only 1 running:

image1276×316 11.7 KB

Took my a while, but I didn’t realise the default was a concurrent setting of 1 in the config.toml, so that’s the problem.

Suprised the system doesn’t just default to “as many runners as you have registered”