ENV: SaaS Gitlab.com + self-hosted runners
We are suddenly seeing an issue where our self-hosted runners are being delegated jobs that are not tagged with tags the runner has assigned to it.
We have multiple runners each registered with a shell and docker executor type. Each type has different tags (docker,gpu) and (shell,gpu). Sporadically some of our CI jobs tagged as (docker,gpu) are picked up by a shell executor and naturally fail. We are usually able to retry the job until it is picked up by a docker executor running with matching tags and succeeds.
Currently on 14.10.1 on all runners. They are shared and being used within our private org. I have attached a scrubbed config file, which was working fine for over 2 months until we suddenly began seeing this behavior.
To add to this, it also appears that the CI/CD Runner page Job count for the shell runners does not increment when it executes a job during an event of the aforementioned undesired behavior. Furthermore, pausing and even Deleting the runners from the UI also does nothing, the paused/removed shell runners will still pickup jobs destined for a runner tagged with docker,gpu.
concurrent = 5
check_interval = 0
[session_server]
session_timeout = 1800
[[runners]]
name = "hostname"
url = "https://gitlab.com/"
token = token
executor = "docker"
[runners.custom_build_dir]
[runners.cache]
[runners.cache.s3]
[runners.cache.gcs]
[runners.cache.azure]
[runners.docker]
tls_verify = false
image = "docker:20.10.16"
privileged = true
disable_entrypoint_overwrite = false
oom_kill_disable = false
disable_cache = false
volumes = ["/certs/client", "/cache"]
shm_size = 0
pull_policy = "always"
gpus = "all"
[[runners]]
name = "hostname-shell"
url = "https://gitlab.com/"
token = token
executor = "shell"
[runners.custom_build_dir]
[runners.cache]
[runners.cache.s3]
[runners.cache.gcs]
[runners.cache.azure]