I’m testing out Auto-DevOps, should I deploy a new kubernetes cluster for each new project? I’ve tried reusing the old cluster, but it does not detect that helm, ingress, and prometheus are already setup there.
You need to setup at group level otherwise you need a kubernetes cluster per project. I also did this mistake and learned the hard way, in particular it is a bit hard to remove the one click installs. So had to create a new cluster and add it as group, you have less one click available but can install using helm directly.
I also tried to use kubernetes with more than one project.
But i managed to reuse the same cluster for a meta-project, as it runs on bare-metal in a distant datacenter.
I did not set up prometheus in my first try, but certbot and ingress. Maybe the commands i used to cleanse the cluster can be some help for you:
kubectl delete ns gitlab-managed-apps kubectl delete customresourcedefinitions.apiextensions.k8s.io certificates.certmanager.k8s.io clusterissuers.certmanager.k8s.io issuers.certmanager.k8s.io kubectl delete clusterrole ingress-nginx-ingress kubectl delete clusterrolebindings.rbac.authorization.k8s.io certmanager-cert-manager ingress-nginx-ingress tiller-admin
You will have to find out, what might be left over from
prometheus after this.
kubectl get clusterroles kubectl get customresourcedefinitions kubectl get clusterrolebindings
If you failed to remove something and your next prometheus installation fails, fear not.
Just check the log of the installation pod, remove the all the tiller stuff from your cluster (adding what you previously missed) and reinstall again.