Separate runners on different K8s clusters with tags

Problem to solve

Hello folks,

I have a problem when deploying GitLab Runner on multiple Kubernetes clusters. I deploy two GitLab Runner using Helm Chart to two different Kubernetes cluster (2 GitLab Runners use the same Runner Token that I got from GitLab Runner UI). I want my job will trigger the runner on the specific cluster by respective tags.

Steps to reproduce

Step 1: I create a New Runner Group with 2 tags called cluster-A and cluster-B:

Step 2: I create a file values-A.yaml:

gitlabUrl: https://gitlab.example.com

runnerToken: "TOKEN"
runnerRegistrationToken: ""

concurrent: 15

tags: "cluster-A"

rbac:
  create: true

runners:
  config: |
    [[runners]]
      name = "sre-devops-runner-A"
      executor = "kubernetes"
      [runner.kubernetes]
        tags = "cluster-A"
        image = "alpine:latest"

And values-B.yaml:

gitlabUrl: https://gitlab.example.com

runnerToken: "TOKEN"
runnerRegistrationToken: ""

concurrent: 15

tags: "cluster-B"

rbac:
  create: true

runners:
  config: |
    [[runners]]
      name = "sre-devops-runner-B"
      executor = "kubernetes"
      [runner.kubernetes]
        tags = "cluster-B"
        image = "alpine:latest"

Step 3: I install GitLab Runner on cluster A by using Helm Chart with file values-A.yaml and GitLab Runner on cluster B by using Helm Chart with file values-B.yaml.

Step 4: When I created my job in CI/CD pipeline with tags cluster-A or tags cluster-B, the runner on random cluster will be triggered:

build:
  stage: build
  image:
    name: docker:24.0.5
  tags:
    - cluster-B
  script:
    - echo "Test"

How can I trigger the runner on cluster-B when tags is cluster-B and trigger the runner on cluster-A when tags is cluster-A?

Version:

My GitLab version is 16.6

Many thanks!