on-prem CE 14.6.1 via docker instance using kubernetes CI/CD runner executor, cluster impl is K3S 1.24, runner is installed via Helm chart 0.36.1.
I’ve recently implemented a backup of our K3S cluster - each server, agent node in our runner cluster. That sequence is: (on each node) - drain, stop k3s service, backup, restart k3s server, uncordon the node.
This, of course, evicts the current runner pods which is expected. Upon restart+uncordon, new pods => new runners because I’ve used the runnerRegistrationToken via a k8s Secret, the default value of unregisterRunner == true (from the 0.36.1 values in the chart) per the recommendation in the values file.
My question is: is there a downside to having this happen 3 times weekly? (could even go to daily backup but the key is to have backups for DR - our earlier k3s cluster did encounter a non-recoverable dslite corruption, had to reinstall the cluster). I’ve also written a script that will identify obsolete runners and via the REST API, delete these instances. Current strategy is to delete runners that are > 7 days since last contacted & who description doesn’t match any current pods in the namespace being used for the runners.
Should I be using the runnerToken on restarted Pods rather than registering new runners? Upside, downside of each approach? The upside of using the current runnerRegistrationToken is no need to update the installed chart’s values but if there’s a downside, I’d rather go with the approach that minimizes any costly downsides…
Other questions: I could change the deployed Helm chart’s values to use the runnerToken value but my understanding is this is the authentication token obtained when the runnerRegistrationToken is successfully used to register the runner initially. Since this is done by the runner’s pod at startup & the pod’s logs don’t include this authentication token (the above 0.36.1 values file even mentions " we don't yet have docs for that, but we want to use existing token", I don’t have an auth value to utilize.
Would using the auth token cause the new pod instance to re-use the same runner? And am I correct that Token= portion of the output of gitlab-runner list is the authentication token for that runner? As the displayed token is NOT the registration token used in the secret provided to the chart. While the same registration token can be used to register multiple runners (2 different charts, using different tags, etc.), I’d expect the gitlab-runner list => Token value to be unique within the gitlab instance. So far, that’s what I’m seeing. And this token value shows up in the ~/.gitlab-runner/config.toml file for the property “token”.
The name of the runner is the name of the pod - is the runner naming the pod or is the pod (via registration) naming the runner?
Fair for me to assume that evicting a runner’s pod does not result in the runner being unregistered (that happens automatically only when the chart’s configuration is updated?), that’s why it shows up as off-line?
MTIA for any feedback, suggestions, insights.