Gitlab-runner check job infrequently

GitLab CI/CD
1

Gitlab runner status:
Runtime platform arch=amd64 os=linux pid=2866 revision=54944146 version=13.10.0
gitlab-runner: Service is running!

Issue: (This issue starts to happen yesterday.)

  • there are a few “shell” [[runners]] in config.toml
  • the runner can execute jobs
  • the runner can check jobs for all the “runner” fastly once I restart the gitlab-runner, it can finish checking for all “runners” within 1 second
  • but after the restart, runner check jobs for each “runner” every 50 seconds, so a job may stuck in pending status for a long time

this is the syslog for gitlab-runner:
Mar 31 19:21:38 s-hdp-vre-h007 gitlab-runner[2889]: #033[37;1mDialing: tcp git.—.org:443 … #033[0;m
Mar 31 19:22:28 s-hdp-vre-h007 gitlab-runner[2889]: #033[37;1mChecking for jobs… nothing #033[0;m #033[37;1mrunner#033[0;m=-tXfc1Vw
Mar 31 19:22:28 s-hdp-vre-h007 gitlab-runner[2889]: #033[37;1mDialing: tcp git.—.org:443 … #033[0;m
Mar 31 19:23:18 s-hdp-vre-h007 gitlab-runner[2889]: #033[37;1mChecking for jobs… nothing #033[0;m #033[37;1mrunner#033[0;m=AQ_7KRMA
Mar 31 19:23:18 s-hdp-vre-h007 gitlab-runner[2889]: #033[37;1mDialing: tcp git.—.org:443 … #033[0;m
Mar 31 19:24:08 s-hdp-vre-h007 gitlab-runner[2889]: #033[37;1mChecking for jobs… nothing #033[0;m #033[37;1mrunner#033[0;m=pLctsvC6
Mar 31 19:24:08 s-hdp-vre-h007 gitlab-runner[2889]: #033[37;1mDialing: tcp git.—.org:443 … #033[0;m
Mar 31 19:24:58 s-hdp-vre-h007 gitlab-runner[2889]: #033[37;1mChecking for jobs… nothing #033[0;m #033[37;1mrunner#033[0;m=mQk2kh3e
Mar 31 19:24:58 s-hdp-vre-h007 gitlab-runner[2889]: #033[37;1mDialing: tcp git.—.org:443 … #033[0;m
Mar 31 19:25:48 s-hdp-vre-h007 gitlab-runner[2889]: #033[37;1mChecking for jobs… nothing #033[0;m #033[37;1mrunner#033[0;m=MYfi2MmQ
Mar 31 19:25:48 s-hdp-vre-h007 gitlab-runner[2889]: #033[37;1mDialing: tcp git.—.org:443 … #033[0;m
Mar 31 19:25:48 s-hdp-vre-h007 gitlab-runner[2889]: #033[37;1mFeeding runners to channel #033[0;m #033[37;1mbuilds#033[0;m=0
Mar 31 19:26:38 s-hdp-vre-h007 gitlab-runner[2889]: #033[37;1mChecking for jobs… nothing #033[0;m #033[37;1mrunner#033[0;m=1ZyRGML-
Mar 31 19:26:38 s-hdp-vre-h007 gitlab-runner[2889]: #033[37;1mDialing: tcp git.—.org:443 … #033[0;m
Mar 31 19:26:38 s-hdp-vre-h007 gitlab-runner[2889]: #033[37;1mChecking for jobs… nothing #033[0;m #033[37;1mrunner#033[0;m=QEhHzSuf
Mar 31 19:26:38 s-hdp-vre-h007 gitlab-runner[2889]: #033[37;1mDialing: tcp git.—.org:443 … #033[0;m
Mar 31 19:26:38 s-hdp-vre-h007 gitlab-runner[2889]: #033[37;1mChecking for jobs… nothing #033[0;m #033[37;1mrunner#033[0;m=X8kYT_aj
Mar 31 19:26:38 s-hdp-vre-h007 gitlab-runner[2889]: #033[37;1mDialing: tcp git.—.org:443 … #033[0;m
Mar 31 19:26:38 s-hdp-vre-h007 gitlab-runner[2889]: #033[37;1mChecking for jobs… nothing #033[0;m #033[37;1mrunner#033[0;m=yww8Ew76
Mar 31 19:26:38 s-hdp-vre-h007 gitlab-runner[2889]: #033[37;1mDialing: tcp git.—.org:443 … #033[0;m
Mar 31 19:26:38 s-hdp-vre-h007 gitlab-runner[2889]: #033[37;1mChecking for jobs… nothing #033[0;m #033[37;1mrunner#033[0;m=qKVfHAM2
Mar 31 19:26:38 s-hdp-vre-h007 gitlab-runner[2889]: #033[37;1mDialing: tcp git.—.org:443 … #033[0;m
Mar 31 19:26:38 s-hdp-vre-h007 gitlab-runner[2889]: #033[37;1mChecking for jobs… nothing #033[0;m #033[37;1mrunner#033[0;m=71NEdfsF
Mar 31 19:26:38 s-hdp-vre-h007 gitlab-runner[2889]: #033[37;1mDialing: tcp git.—.org:443 … #033[0;m
Mar 31 19:27:28 s-hdp-vre-h007 gitlab-runner[2889]: #033[37;1mChecking for jobs… nothing #033[0;m #033[37;1mrunner#033[0;m=jXsX7XUp
Mar 31 19:27:28 s-hdp-vre-h007 gitlab-runner[2889]: #033[37;1mDialing: tcp git.—.org:443 … #033[0;m
Mar 31 19:27:28 s-hdp-vre-h007 gitlab-runner[2889]: #033[37;1mChecking for jobs… nothing #033[0;m #033[37;1mrunner#033[0;m=6dpsZMrD
Mar 31 19:27:28 s-hdp-vre-h007 gitlab-runner[2889]: #033[37;1mDialing: tcp git.—.org:443 … #033[0;m
Mar 31 19:28:18 s-hdp-vre-h007 gitlab-runner[2889]: #033[37;1mChecking for jobs… nothing #033[0;m #033[37;1mrunner#033[0;m=q9qaMWMK
Mar 31 19:28:18 s-hdp-vre-h007 gitlab-runner[2889]: #033[37;1mDialing: tcp git.—.org:443 … #033[0;m
Mar 31 19:29:08 s-hdp-vre-h007 gitlab-runner[2889]: #033[37;1mChecking for jobs… nothing #033[0;m #033[37;1mrunner#033[0;m=HyMpUrTu
Mar 31 19:29:08 s-hdp-vre-h007 gitlab-runner[2889]: #033[37;1mDialing: tcp git.—.org:443 … #033[0;m
Mar 31 19:29:08 s-hdp-vre-h007 gitlab-runner[2889]: #033[37;1mChecking for jobs… nothing #033[0;m #033[37;1mrunner#033[0;m=MHVbHwiy
Mar 31 19:29:08 s-hdp-vre-h007 gitlab-runner[2889]: #033[37;1mDialing: tcp git.—.org:443 … #033[0;m

  • What are you seeing, and how does that differ from what you expect to see?
  • Consider including screenshots, error messages, and/or other helpful visuals
  • What version are you on? Are you using self-managed or GitLab.com?
    • GitLab (Hint: /help):
    • Runner (Hint: /admin/runners):

Here is the global config for config.toml:
concurrent = 1
check_interval = 1
log_level = “debug”

[session_server]
session_timeout = 1800

  • Add the CI configuration from .gitlab-ci.yml and other configuration if relevant (e.g. docker-compose.yml)
    CI configuration is fine.

  • What troubleshooting steps have you already taken? Can you link to any docs or other resources so we know where you have been?

  1. remove runner from Gitlab project and add runner again
  2. uninstall gitlab-runner on the VM and install the latest version
  3. restart the VM running gitlab-runner

Thanks for taking the time to be thorough in your request, it really helps! :blush: