Dead Runner Still Dangling

At my company I am the one primarily responsible(unofficially) for runner administration. I have created an internal platform for users to deploy AWS autoscaling docker runners, where there is a runner manager/coordinator that spins up executors to process jobs.

When a user deploys a runner, a coordinator instance is deployed and it registers to the group or project for which the user provided an access token. When the coordinator is destroyed using the automation platform, or some other reason when the lifecycle hook has time to kick in, a command is executed on the instance to unregister the runner from gitlab. This way, when a user looks in that repository’s CI/CD setting, that runner is no longer listed.

The problem is that during development of this platform, many runners were provisioned, registered, and then destroyed, all without sending the unregister command. There are also times when a runner gets terminated unexpectedly and the runner doesn’t get unregistered for some reason.

Our Gitlab is hosted by Gitlab.com, and we have all of our repos and groups under one large umbrella group. There were dozens of runners that got registered to this top-level group that got destroyed before they were unregistered. Now we have dozens of unavailable runners showing up in every repository’s CI/CD settings with a red dot next to them in both the “Other Available Runners” section underneath “Specific Runners”, as well as in the “Group Runners” section.

I have maintainer privileges on many repos in the company group. I would like to know how I can go about removing these defunct runners from these lists. They no longer exist, so there is no way to execute an unregister command on them. I assume that if I got high level permissions on the top-level company group I could accomplish this, but is there a way to do this without requesting god level permissions for the entire company Gitlab? I can probably get them, but I would rather do this in a way that doesn’t involve me having to bug management for such high privileges and explain why I need them.

If the only way to accomplish this is to get elevated privileges, which privileges do I need specifically? Will maintainer suffice, or do I need to have an owner role? Can I clear these runners from the top level group and expect it to remove them from child repos, or do I have to remove them from each repo individually?

Thanks in advance for any help/ideas on how to take care of this. It isn’t a pressing issue, but it looks messy and getting this taken care of will make the idea of developers provisioning their own runners more welcoming.

Group owners can manage runners. You’ll see them in Settings > CI/CD > Runners, and can remove them in the list using the X icon.

Example from a group:


1 Like

Thank you! I was worried that only owners were allowed to perform that action. Is there a way for an owner to grant me temporary owner privileges to clean the runners up, or will they have to assign me the owner role manually and then revoke it when I am finished?

If there is no simple way to do that, is it possible to remove dangling runners with the API? I am thinking that if this is possible via the Gitlab API, then an owner can create an automatically expiring API.

Thanks again for the response.

I know that there is an expiry date for inviting users, but afterwards they are removed from a group. I don’t think that this exists for elevated permissions with an expiry date to restore the old level (could be an interesting feature request). For your case, a manual change to owner and then revoke could work. Or you’ll pair with an owner and do it over screensharing, if processes require that owners cannot be assigned.

Runners API | GitLab allows to list all group runners, and you should be able to delete them using the ID. Using a group access token, which you’ll need to obtain from your group owners. This token needs api scope and should have an expiry date. Group access tokens | GitLab

1 Like

Great, thank you for answering my questions. You folks at Gitlab are the best!

1 Like