Discussion: Runner Tag Management Strategy in Kubernetes - Seeking Feedback

Infrastructure as Code & Cloud Native
,
1

Problem to solve

We run GitLab Runners in Kubernetes with the following workflow:

  1. Deploy runners with initial tags (e.g., testing)
  2. Run e2e tests against these runners to verify their health
  3. Upon successful tests, update the runner tags to make them available for developer jobs

Questions for Discussion

  1. Is this a valid approach for managing runner lifecycle in Kubernetes?
  2. Would moving to immutable runners be a better pattern? (i.e., create new runners with final tags, test them, and destroy/replace old ones)
  3. What are the pros/cons of each approach that other teams have experienced?
  4. Are there existing tools/patterns we should consider for either approach?

Looking forward to community insights and experiences with similar setups.

Context

  • Environment: Kubernetes
  • Using: gitlab-runner Kubernetes executor
  • Looking to understand best practices for runner lifecycle management

Versions

Please select whether options apply, and add the version information.

  • Self-managed
  • GitLab.com SaaS
  • Dedicated
  • Self-hosted Runners

Versions

  • GitLab Runner:
    17.4.0