I have 1 gitlab-runner instance on my VPS. Whenever I cancel any previous job (for example, when I realize I don’t want something to run) all following jobs stay in pending way too long. That is, there are no running jobs anymore, yet there are pending ones. It takes about 5 minutes for the next job to start running and get out of pending mode.
Any ideas why that is?
I experienced this too. My gitlab-runner was a shell executor. When it runs a job it creates a process tree like that we can see with the pstree command. Here’s an example from my use:
$ pstree -p | grep gitlab-runner
|-gitlab-runner(1762)-+-su(7041)---bash(7050)---bash(7087)---make(7088)---make(7130)---make(7184)---python(7208)---scp(7233)---ssh(7234)
D0han in the IRC channel informed me that hitting the Cancel button only kills the su process which is not passing on the SIGKILL signal to its children.
$pstree -p | grep "bash.*bash.*make.*make"
|-bash(7050)---bash(7087)---make(7088)---make(7130)---make(7184)---python(7208)---scp(7233)---ssh(7234)
You can see by the process ID’s that this is the same process tree except that the gitlab-runner and su processes are no longer running.
What I’ve experienced is that my new jobs will remain pending until this entire process tree has completed. D0han indicated that other types of gitlab-runner executors have a different behavior when it comes to cancelling, so I’m going to move on to a different runner type.
Looks like there has been a ticket for this for a couple years now: https://gitlab.com/gitlab-org/gitlab-runner/issues/1662.