Session terminated, killing shell... ...killed when restarting runner

Hi Gitlab community,

I am seeing a very odd behaviour with my CI pipeline which deploys my runners.

A bit of background first, I am running Gitlab CE on a Vsphere hosted VM. I installed a single runner on the gitlab server to service the project that deploys runner servers. This is tag-locked so that the project does not use any other runner but the local one.

I set up a CI pipeline which has three stages - terraform, ansible and clean. The first runs terraform to create VMs. These are then provisioned by the second stage using ansible and it is here that I am running into an issue.

My ansible customises the gitlab-runner host VM and installs the runner software, registers runners (shell) and then customises the config.toml followed by a gitlab-runner restart to re-read the config.toml. It is here that the problem occurs. This runs fine when run manually but when the local gitlab-runner is running it I get “session terminated, killing shell… …killed.” in the pipeline output after the restart. Perusing the logs on the gitlab-runner host ansible is running still and completes.

It almost looks like restarting the runner service on the gitlab-runner host is somehow affecting the local runner on the gitlab host.

Any suggestions?

I have a hunch about what is causing this issue of mine. If I execute it with existing runners this issue might occur.

I thought tagging the project with tag “x” and tagging the runner with “x” would mean only runners with tag “x” would execute the jobs for that project. I was wrong. Can anyone advise how to restrict a project to only use one specific runner ie not one of the shared runners?

The automation creates the runners on the gitlab-runner server, once they are up they are available in gitlab. If the job is executed with existing runners available then the ansible gets executed by one of the shared runners and when the restart occurs it kills it hence the error.

So the way to solve it is to lock it to one and only one runner and not allow shared runners to execute it.