Not able to register hosted runners, getting HTTP status 400

Failing to register self hosted gitlab runners on kubernetes. We had been running these for a while before they stopped working abruptly.
(gitlab runner tag docker image gitlab-runner:alpine-v14.1.0)

configmap for the deployment is attached for reference.

k8s event:
Warning Unhealthy 3m18s (x68 over 17m) kubelet Readiness probe failed

Logs from gitlab-runner deployment (prefixed by pod name)

gitlab-gitlab-runner-7f6d67fbd4-fnsdg gitlab-gitlab-runner WARNING: Running in user-mode.
gitlab-gitlab-runner-7f6d67fbd4-fnsdg gitlab-gitlab-runner WARNING: The user-mode requires you to manually start builds processing:
gitlab-gitlab-runner-7f6d67fbd4-fnsdg gitlab-gitlab-runner WARNING: $ gitlab-runner run
gitlab-gitlab-runner-7f6d67fbd4-fnsdg gitlab-gitlab-runner WARNING: Use sudo for system-mode:
gitlab-gitlab-runner-7f6d67fbd4-fnsdg gitlab-gitlab-runner WARNING: $ sudo gitlab-runner...
gitlab-gitlab-runner-7f6d67fbd4-fnsdg gitlab-gitlab-runner
gitlab-gitlab-runner-5b85dcd447-bgltb gitlab-gitlab-runner Registration attempt 14 of 30
gitlab-gitlab-runner-5b85dcd447-bgltb gitlab-gitlab-runner Runtime platform arch=amd64 os=linux pid=261 revision=8925d9a0 version=14.1.0
gitlab-gitlab-runner-5b85dcd447-bgltb gitlab-gitlab-runner WARNING: Running in user-mode.
gitlab-gitlab-runner-7f6d67fbd4-fnsdg gitlab-gitlab-runner ERROR: Registering runner... failed runner=jAQ_rW_Z status=400 Bad Request
gitlab-gitlab-runner-7f6d67fbd4-fnsdg gitlab-gitlab-runner PANIC: Failed to register the runner. You may be having network problems.
gitlab-gitlab-runner-5b85dcd447-bgltb gitlab-gitlab-runner WARNING: The user-mode requires you to manually start builds processing:
gitlab-gitlab-runner-5b85dcd447-bgltb gitlab-gitlab-runner WARNING: $ gitlab-runner run
gitlab-gitlab-runner-5b85dcd447-bgltb gitlab-gitlab-runner WARNING: Use sudo for system-mode:
gitlab-gitlab-runner-5b85dcd447-bgltb gitlab-gitlab-runner WARNING: $ sudo gitlab-runner...
gitlab-gitlab-runner-5b85dcd447-bgltb gitlab-gitlab-runner
gitlab-gitlab-runner-5b85dcd447-bgltb gitlab-gitlab-runner ERROR: Registering runner... failed runner=jAQ_rW_Z status=400 Bad Request
gitlab-gitlab-runner-5b85dcd447-bgltb gitlab-gitlab-runner PANIC: Failed to register the runner. You may be having network problems.

the problem fixed itself after a few days wtf gitlab :man_shrugging: