Kubernetes Integration "Something went wrong while installing Helm Tiller"

I’m trying to connect gitlab to kubernetes. But everytime I set it up and try to install Helm from integration page I got this error:
Something went wrong while installing Helm Tiller
Can’t start installation process

I went throught all /var/log/gitlab/ logs, but I can’t see anything in there.
In /var/log/gitlab/gitlab-rails/production.log every GET request shows Completed 200 OK in … ms …
Nothing in kubernetes logs as well.

We have our production kubernetes only one week so maybe there is something wrong, but everything else is working there. I tried to set it up with and without CA cert and namespace filled on integration page with the same result.
I have installed clean kubernetes and gitlab just with our certificates on libvirt to avoid conflicts with anything else but I still can’t solve it.

Any clues how to find any usefull information, how to test integration or where to look for error?

3 Likes

Got the same problem with a cluster running in GKE. However, if I remove the cert the installation process reports Kubernetes error: SSL_connect returned=1 errno=0 state=error: certificate verify failed. But it seems like if I write anything in the cert field it gets accepted.

Same problem for me. I’m not able to see any details on logs.
Can we install helm using another method?

Hi folks - GitLabber here. Sorry to here you are experiencing trouble with installing Helm. I’ve logged an issue to update the error message and docs: https://gitlab.com/gitlab-org/gitlab-ce/issues/45939

The goal of the one-click installs for the cluster integration is to eliminate manual set up. While there are bugs in the automated process, it is still possible to manually install helm. It requires you to set up kubeclt on your local machine to connect to your Kubernetes cluster. There are instructions here:
https://docs.gitlab.com/ee/topics/autodevops/quick_start_guide.html#connect-to-kubernetes-cluster

3 Likes

Hi

I had the same error with minikube and GitLab CI/CD when adding a kubernetes cluster.

In my case I inspected the error manually with openssl tools like so:

$ openssl s_client -showcerts -connect 192.168.100.10:8443 < /dev/null &> apiserver.crt

In the first lines of output, the apiserver's certificate shows that the Subject and the Issuer.

Certificate chain
 0 s:/O=system:masters/CN=minikube
   i:/CN=minikubeCA

We need to get the issuer’s (root) certificate with Common Name minikubeCA. I found such certificate on ~/.minikube/ca.crt in the minikube host. That’s the one you need to paste in the GitLab Kubernetes cluster integration page:


Extra section

To check all the certificates on the minikube directory use this command and look for a certificate whose output looks like this:

$ find ~/.minikube/ \( -iname '*.crt' -o -iname '*.pem' \) -ls -exec openssl x509 -in {} -noout -subject -issuer \;

	...

123456      4 -rw-r--r--   1 tonejito   users       1066 May 10 18:00 ~/.minikube/ca.crt
issuer= /CN=minikubeCA
subject= /CN=minikubeCA

	...

Use OpenSSL to check if the CA certificate works:

$ openssl verify -verbose -CAfile ca.crt apiserver.crt
apiserver.crt: OK

I’ve installed another single-node “cluster” to test GitLab integration with Kubernetes following the official guide with kubeadm [1].

According to the official documentation, the API URL is only https://hostname:port without trailing slash.

First, I listed the secrets as usual:

$ kubectl get secrets
NAME                           TYPE                                  DATA      AGE
default-token-tpvsd            kubernetes.io/service-account-token   3         2d
k8s-dashboard-sa-token-XXXXX   kubernetes.io/service-account-token   3         1d

Then I got the CA certificate directly from the JSON output via jq with a custom selector:

$ kubectl -o json get secret k8s-dashboard-sa-token-XXXXX | jq -r '.data."ca.crt"' | base64 -d - | tee ca.crt
-----BEGIN CERTIFICATE-----
MIICyDCCAbCgAwIBAgIBADANBgkqhkiG9w0BAQsFADAVMRMwEQYDVQQDEwprdWJl
	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...
FT55iMtPtFqAOnoYBCiLH6oT6Z1ACxduxPZA/EeQmTUoRJG8joczI0V1cnY=
-----END CERTIFICATE-----

After this, I could verify the CA certificate as usual:

$ openssl x509 -in ca.crt -noout -subject -issuer
subject= /CN=kubernetes
issuer= /CN=kubernetes

$ openssl s_client -showcerts -connect 192.168.100.20:6443 < /dev/null &> apiserver.crt

$ openssl verify -verbose -CAfile ca.crt apiserver.crt
apiserver.crt: OK

Happy hacking!

2 Likes

I cannot find instructions to manually install helm there. Have they been (re-)moved?

1 Like

Links to official docs for installing kubeclt and helm: https://docs.gitlab.com/ee/install/kubernetes/preparation/tools_installation.html#doc-nav

1 Like

As far as I can see, these docs refer to Installing GitLab on Kubernetes, which is not the same as the Kubernetes integration we’re talking about here, is it?

The Kubernetes integration provides a one-click install of Helm, Ingress and Prometheus in a dedicated gitlab-managed-apps namespace in the Kubernetes cluster (see docs).

If the one-click install fails, how can I install Helm in such a way that it can be recognized and utilized by GitLab thereafter (i.e. appears as “Installed” in the Applications sections of the Cluster page in GitLab)?

3 Likes

Seeing this today. I can deploy projects to kubernetes but I cannot install gitlab runner because I get "
Something went wrong while installing Helm Tiller

  • Kubernetes error."
    How can I install Helm Tiller so that Gitlab will see it. The kubernetes cluster is in use by several gitlab projects using autodevops which all deploy successfully.

Can you paste the kubectl commands here? I cannot find any mention of kubectl on the page you provided.

My cluster already has helm and tiller, but it wont connect I get the same error, I just want gitlab to build and deploy to my cluster.

Guys, has anyone really really solved this issue while using gitlab.com and adding an existing cluster?

Since, I am using gitlab.com, I don’t have access to the logs and I am seeing the same helpful error message on the UI.

ANY sort of help will be very appreciated since technically, as a user, I can’t understand and fix the issue unless I know what the issue is.

After a huge amount of research I guess that this problem has no solution. I’ve tried dozens of solutions but none of them works…

  1. Use gitlab.com to integrate the cluster
  2. Use localhost gitlab
  3. Use CA certificate
  4. Don’t use CA certificate
  5. Re-install the entire local gitlab
  6. Re-install the minikube

None of the above tries works to me

There are a couple of things that might help.

  1. The error reporting has been improved a little so you should now get at least an error code like 401 or 403 back. 401 may mean a problem with your token. Make sure you’re following the instructions for how to get the token. You cannot get the token from the web admin pages in GKE, must use kubectl. The token should not have any spaces in it. I’ve seen spaces put in when copying/pasting from a web terminal. All the tokens I’ve seen begin with the characters ey, if yours doesn’t check to make sure you’re using base64 --decode correctly. 403 may mean that the token you have has insufficient permissions in your cluster, try enabling Legacy Authorization in a GKE cluster and turning off RBAC when adding the cluster to GitLab. If you must have RBAC, you’ll need to create a serviceaccount in your cluster that has clusteradmin privileges.

  2. If your cluster already has the gitlab-managed-apps namespace, remove it with kubectl. If it exists it will cause the installation of Helm Tiller to fail.

  3. The only Helm Tiller that works for the integration is the one installed into gitlab-managed-apps. If there are other Helm Tillers, I recommend removing them.

These are the ones that come to mind that have worked with most of the people I’ve guided through this.

~Steevo

3 Likes

Initially this worked but after it did not… i get the following message

$HELM_HOME has been configured at /root/.helm.
Error: error installing: deployments.extensions is forbidden: User “system:serviceaccount:gitlab-managed-apps:default” cannot create deployments.extensions in the namespace “gitlab-managed-apps”

Any idea why this happens?

Thank you a lot! That GKE console put some tricky spaces in tokens on copying. :grinning:

1 Like

I am adding an existing kubernetes cluster to gitlab, while installing helm tiller am getting the error
something went wrong while installing Helm Tiller

Below is the log from gitlab server.

{“severity”:“ERROR”,“time”:“2019-03-07T07:55:06.874Z”,“correlation_id”:“dIpWGiwpO51”,“exception”:“Kubeclient::HttpError”,“error_code”:null,“service”:“Clusters::Applications::InstallService”,“app_id”:3,“project_ids”:[343],“group_ids”:,“message”:“SSL_connect returned=1 errno=0 state=error: certificate verify failed (unable to get local issuer certificate)”}

please let know how to resolve the issue

If you are getting a Kubernetes Error 401 it means your Token is incorrect as it was in my case. Here I was using a token from an API tab in the Dashboard BUT should really create the Token by command line like recommended. After generating a user and a token using kubectl commands and changing the Token the installation of Helm worked perfectly.
Hope it helps!

The “Helm Tiller” install is a no go. We will be sticking with GitLab Runner.

GitLab Auto DevOps is garbage if it cannot even install successfully or provide meaningful errors.

Wasting everyone’s time spinning up k8s to choke on the integration does not inspire goodwill.