Orphaned apps in gitlab-managed-apps namespace

I upgraded our gitlab server to 11.6 and decided to use the new group Kubernetes feature.
The group k8s only allows you to install helm, nginx and cert-manager from gitlab.
But our runner was previously installed using gitlab in gitlab-managed-apps.

I used to be able to use the command helm list --tiller-namespace gitlab-managed-apps from my laptop and manage those apps if needed.
But now I get a Error: transport is closing which probably means that tiller in that namespace is now configured to use TLS. And Gitlab must be managing that now.

So how in the world am I supposed to cleanup the GitLab abandoned apps in that namespace so I can manage them myself?
I’ve already created my own gitlab runner deployment in the cluster using helm, but there is no way to delete the one in the gitlab-managed-apps namespace.

Hey! I have the exact same problem, did you manage to find a solution? I have a failed release that I need to delete, and now I can’t deploy anymore.

Hey @tinco no I have not found a solution yet. So far I’ve just been ignoring the orphaned apps in that namespace.
I’m guessing the solution will have something to do with finding the the helm key files being used by gitlab.

I just made some progress… I just deleted the tiller service, and then recreated it from the commandline. Don’t do that on your production just yet though, because now I have a different error having to do with RBAC

OK yeah I got it to work, now helm ls works again. I had to specify the --service-account to be something that had the right access.

Weird thing though, it’s only showing releases in the gitlab-managed-apps namespace, but my apps are deployed in another namespace that it doesn’t show the releases for…

@tinco I’ve just had the same experience, unpacked the secrets and connected to Helm using the same version as Tiller (installed via Gitlab’s K8s integration) as so:

helm list --tiller-connection-timeout 30 --tls --tls-ca-cert tiller-ca.crt --tls-cert tiller.crt --tls-key tiller.key --tiller-namespace kube-system --all --tiller-namespace gitlab-managed-apps

But the only apps in there are the ones from K8s integration, not the release of my application. I’ve made a change to my chart name but I can’t work out how to remove the old release, Gitlabs docs don’t mention any way of removing their autodevops installs apps and it’s won’t upgrade now the chart name is being changed.

Any progress?

Hey Mark! Yes the apps are actually each in their own namespace, so you have to specify that in your ls command. You got much further than I did, actually managing to connect with tls. What wizardry gave you the cert?

Thanks @tinco, I’ll try that now. See wizardry below!

export TILLER_NAMESPACE="gitlab-managed-apps"

kubectl get secrets/tiller-secret -n "$TILLER_NAMESPACE" -o "jsonpath={.data['ca\.crt']}" | base64 -d > tiller-ca.crt
kubectl get secrets/tiller-secret -n "$TILLER_NAMESPACE" -o "jsonpath={.data['tls\.crt']}" | base64 -d > tiller.crt
kubectl get secrets/tiller-secret -n "$TILLER_NAMESPACE" -o "jsonpath={.data['tls\.key']}" | base64 -d > tiller.key

helm list --tiller-connection-timeout 30 --tls --tls-ca-cert tiller-ca.crt --tls-cert tiller.crt --tls-key tiller.key ---all --tiller-namespace gitlab-managed-apps

Hmm, I’m still only seeing the Gitlab Runner’s release, not my own app deployed in a different namespace. At this stage i’m thinking it might just be quicker to rebuild the cluster! The all flag has no effect, and the namespace containing my app gives no results.

Hey Mark, sorry I was not specific enough. The apps actually run in separate tiller namespaces. So with the kube-system namespace you’ll only see the gitlab-managed-apps namespace. I forgot what the naming for the other namespaces was, but they’re specific to each app, and that’s also why the all flag does nothing.

edit: eh wait, I’m seeing in your command you actually specify tiller-namespace twice, could that cause your command to not work?

Nope, i just spotted that but it was taking the last arguments value anyway. If i’ve understood you right, i should be doing:-

--tiller-namespace gitlab-managed-apps --namespace gitlab-managed-apps to see Gitlab’s own applications, and…

--tiller-namespace gitlab-managed-apps --namespace <my applications namespace> to see my applications release? As this variation is empty.

I’m assuming by “separate tiller namespace” you’re not suggesting changing the actual --tiller-namespace flag value? There’s only one tiller pod in the cluster so that wouldn’t work on any others?

I have no idea why it works, but if I recall correctly changing the --tiller-namespace flag is what I did. Just leave the other one off (I think it will default to --all).

It didn’t make any sense to me either, but there’s no gitlab staff on here to explain it. The whole helm aspect of auto devops is very underdocumented in my opinion.

Agreed on the documentation, the inability to remove projects from k8s again, and being able to remove the Gitlab apps themselves is sorely missing (or completely undocumented) to the point I’m going to rebuild the cluster one last time but otherwise revert to non-autodevops scripts again. Just too brittle if you even remotely step out of the ABC workflows.

Thanks for your help sir.

Using this thread i was able to connect to tiller, bud i can’t see any deployments in namespaces outside of gitlab-managed-apps.

I solved my problem with connecting to other namespaces.

Hey! I have the exact same problem, did you manage to find a solution? I have a failed release that I need to delete, and now I can install mod apk technifiser.