Cancelling Job doesnt free runner

I seem to be having problems on my runners - when I cancel pipeline jobs the actual job continues running on the runners. I have to manually log into the runner and kill the job.
How does gitlab attempt to kill the job? When I look at gitlab-runner verify, I get the following:

ERROR: Verifying runner... is removed               runner=vcofGRE8
Verifying runner... is alive                        runner=N1TCzZhs
FATAL: Failed to verify runners

I think this might be the same issue I have been experiencing. My gitlab-runner was a shell executor. When I hit Cancel gitlab indicates that the job has been cancelled, but using top on the test system indicates the gitlab-runner user is still running the commands from the job. The runner also won’t pick up any new jobs for me until all the commands from that job are complete.

When the CI system runs a job, it creates a process tree that we can see with the pstree command. Here’s an example from my use:

$ pstree -p | grep gitlab-runner
            |-gitlab-runner(1762)-+-su(7041)---bash(7050)---bash(7087)---make(7088)---make(7130)---make(7184)---python(7208)---scp(7233)---ssh(7234)

D0han in the #gitlab IRC channel on freenode informed me that hitting the Cancel button only kills the su process which is not passing on the SIGKILL signal to its children.

$pstree -p | grep "bash.*bash.*make.*make"
            |-bash(7050)---bash(7087)---make(7088)---make(7130)---make(7184)---python(7208)---scp(7233)---ssh(7234)

You can see by the process ID’s that this is the same process tree except that the gitlab-runner and su processes are no longer running.
What I’ve experienced is that my new jobs will remain pending until this entire process tree has completed. D0han indicated that other types of gitlab-runner executors have a different behavior when it comes to cancelling, so I’m going to move on to a different runner type.

See https://gitlab.com/gitlab-org/gitlab-runner/issues/1662.