Gitlab Runner Registration fail for Group but not specific

GitLab CI/CD
1

Gitlab Runner Registration fail for Group but not specific

Hello guys, I’m having one of the weirdest issue. If any one has any insight it would be great.

Describe your question in as much detail as possible:

  • What are you seeing, and how does that differ from what you expect to see?
    I have an issue with the registration of group runner since last night.
    Here is what I see when I try to register a group runner → 
WARNING: Running in user-mode.                     
WARNING: The user-mode requires you to manually start builds processing: 
WARNING: $ gitlab-runner run                       
WARNING: Use sudo for system-mode:                 
WARNING: $ sudo gitlab-runner...                   
                                                   
ERROR: Registering runner... forbidden (check registration token)  runner=FK_UmJZu
PANIC: Failed to register the runner. You may be having network problems.

Nothing changed. The Kubernetes cluster just shutdown during the night and turned on in the morning. The runners could not register anymore.

  • What version are you on? Are you using self-managed or GitLab.com?

  • What troubleshooting steps have you already taken? Can you link to any docs or other resources so we know where you have been?

I have tried the same configuration of runner in my cluster:

  • Runner in version 0.17.0 fails to register when group registration but works when specific
  • Runner in versions 0.29.0 → Same
  • Runner in a local kubernetes cluster → Same
  • Runner install on my mac → Same

I also refreshed the group token but I’m still having the same issue.

1 Like
2

We have the same issue using the latest gitlab-runner. Using a project-level token works, but using a group-level token the runner fails to register itself.

2 Likes
3

We saw almost the same thing in our cluster with a slightly different error:

ERROR: Registering runner... failed                 runner=TsBsC7Cu status=400 Bad Request
PANIC: Failed to register the runner. You may be having network problems

With both gitlab-runner 13.11.0 and 14.1.0 from respective helm chart v0.29.0 and v0.31.0.

Changing it to project-level token works.

I reported it in Registering runner... failed status=400 Bad Request (#303) · Issues · GitLab.org / charts / GitLab Runner · GitLab, but it is probably not related to the helm chart, so closed that one and created Runner registration fails with group token (#337942) · Issues · GitLab.org / GitLab · GitLab

1 Like
4

Got it to work by using the group token for the group above my group.

Not ideal but it is a workaround.

5

Got it to work by using the group token for the group above my group.

We were already at the top level group.

6

We are seeing exactly the same issue as @roelandvanbatenburg and we also are at the top level group. I tried resetting the group token but it still fails to register itself.

7

Are you using spot instances for your runners ?

8

Yes, we are using spot instances.

So, I tried registering the runner using the Runners API and I got this error message:

{
  "message": {
    "runner_namespaces.base": [
      "Maximum number of ci registered group runners (50) exceeded"
    ]
  }
}

It turns out we have had way more than 50 registered group runners for some time now, so I think this limit may have been introduced on a recent change? Anyway, we are deleting all the old runners as a workaround for now.

9

Hi there, thanks for posting your issue! We recently updated a limit and are investigating if that has impacted Group Runners, see Runner registration fails with group token (#337942) · Issues · GitLab.org / GitLab · GitLab.

As a solution try to remove any unused or old Runners from the “Available Runners” section in Group CI/CD Settings. This will resolve the limit and improve your experience while we investigate.

3 Likes