I want to experiment with CI/CD without using AWS or GKE. I’ve tried following the instructions for “Connect to Existing Cluster”, but I cannot get it to work. It keeps saying the cluster is unreachable. I’ve tried to get Kubernetes inside of Docker Desktop for Windows to work (I think this one is impossible), Minikube on Linux, and Kind on Linux.
Does anyone know of a video that shows someone successfully setting up a simple cluster environment for experimentation on their laptop? I want something that runs hello world so I can experiment with CI/CD concepts like deploy to test and then promote to prod and using feature flags. Every CI/CD video I’ve seen starts with “I’m just going to click this button to get a cluster on Google”. At five minutes in, I’m stuck every time.
Connecting GitLab to existing k8 cluster is fairly easy. You need to use the Kubernetes API and the CA file for your K8 cluster. You need to make sure your cluster is reachable from internet, that means having public IP and port 443 available (or using port-forwarding).
Her are the docs how to prepare k8 cluster for GitLab Adding and removing Kubernetes clusters | GitLab
On Windows laptop you can use VirtualBox to spin up Linux box and install minikube/rke/k3s/kind.
Simple guide for minikube Hello Minikube | Kubernetes
How do I expose the service port? I ran this command from the instructions:
kubectl cluster-info | grep -E ‘Kubernetes master|Kubernetes control plane’ | awk ‘/http/ {print $NF}’
The output was (port 8443 is the default control plane port for minikube): https://192.168.49.2:8443
And this is what I see in my .kube/config file as well
However, when I list open ports:
sudo lsof -i -P -n | grep LISTEN
I don’t know which driver you are use using for minikube. I assume ‘docker’ driver.
Minikube sets up port forwarding and masquerade rules using iptables. If you print out iptables NAT table you can see that one of the 4915* ports are forwarded to container port 8443. And also that there is NAT setup for the 192.168.49.0/24 network.However, ports 4915* are only listening on 127.0.0.1 so they can’t be used.
If you need minikube k8 cluster API to be accessible from outside networks you need to setup port-forwarding from the VM to minikube. Something like this: sudo iptables -t nat -A PREROUTING -p tcp -i YOUR_VM_INTERFACE_NAME --dport 8443 -j DNAT --to-destination 192.168.49.2:8443
With Windows on laptop my setup would be Windows 10 + Ubuntu running in VirtualBox + minikube using docker driver in the Ubuntu box. Ubuntu box has bridged adapter so it has it’s own IP on the local network and the above iptables rule. Final step is to setup port-forward on router to Ubuntu IP port 8443.
Since my original post, I’ve moved from my Windows laptop over to an EC2 instance on AWS. It is Amazon Linux and I’m using minikube on docker. Without the extra VirtualBox layer, I was able to run your command using eth0 as the interface name. I can browse to port 8443 and get a response (403 forbidden, but I know it is serving).
I’ve got connectivity now, but the error message in gitlab.com changed to “There was a problem authenticating with your cluster. Please ensure your CA Certificate and Token are valid.”
I’ve checked my CA Certificate and Service Token at least 10 times. I’ve pasted them into notepad, then recopied them out of notepad into the browser. No extra spaces or line feeds. I still get this error.
When I use postman, I’m able to set up a client certificate using these two files (pulled from .kube/config): - name: minikube user: client-certificate: /home/ec2-user/.minikube/profiles/minikube/client.crt client-key: /home/ec2-user/.minikube/profiles/minikube/client.key
In postman, I can add the crt and key and hit the endpoint. It responds back with a list of valid API endpoints.
I created the yaml file and ran it. The output was: serviceaccount/gitlab created clusterrolebinding.rbac.authorization.k8s.io/gitlab-admin created
I deleted the certificate in postman and instead specified bearer token for my request and pasted the token text into the token field (I assume you don’t need any other usernames or keys to use a token in postman), but I get the 403 forbidden response in postman with the token. Is there something else I need to do to get the token based authentication to work correctly in minikube?
Minikube is not using RBAC by default, so make sure you uncheck that box. Otherwise it should work fine.
Unfortunately, that message is a generic one and won’t help you determine whats really wrong.