[SOLVED] Toggling off "Namespace per environment" on GL managed cluster puts all new review releases in the oldest existing namespace

Infrastructure as Code & Cloud Native
, ,
1

We are using Gitlab.com and have a managed project cluster hosting all our AutoDevOps deployed review environments. At the time of the incident it had roughly 30 active environments and almost 300 namespaces in total (GL has a namespace bloating issue).

I wanted to switch from using a namespace per environment to a single namespace for the whole project so toggled the “Namespace per environment” option off in the cluster settings UI. I also added a prefix to the “Project namespace prefix” setting, for the sake of argument let’s say this is abc which matches the project name in lower case. Based on the docs, it looks like setting a prefix with the “Namespace per environment” toggled off would instead use that prefix as the project wide namespace’s name. Based on the previous example prefix, I would expect the namespace to be abc.

After pushing a new branch, the K8s resources are created in one of the existing empty namespaces from an old environment that had been cleaned up many months ago (and so was otherwise empty). This namespace takes the form abc-12345678-review-feature-ad-7zogby. Looking at the ages of all the namespaces, this was the first one created after initially adding the cluster.

This is not a good outcome as this namespace is incorrectly named for something that contains all our environments and I unexpectedly do not have control over the name of it either.

  • Is this the expected behaviour when toggling off “Namespace per environment” on a long running cluster?

  • Is my expectation that the shared namespace should equal the prefix value correct?

  • How can I resolve this situation now that I have workloads running in the wrong namespace and the expected namespace not having been created?

  • We are not using any AutoDevOps overrides or customisations

  • I have tried logging out the $KUBE_NAMESPACE variable, as this was used in the auto-deploy code to determine the namespace value. This was set to the incorrect old namespace name - e.g. abc-12345678-review-feature-ad-7zogby. I haven’t been able to find where this is set so assume it to be calculated and passed in from GL somewhere.

Any help would be greatly appreciated. Happy to answer any questions you may have.

2

I decided to clear the cluster cache and rerun the build and everything is working as expected. Sorry for adding noise to the forum but I hope this helps someone else.

1 Like