Gitlab KAS. Problem with k8s connection

How to Use GitLab Self-managed
1

Hello there.

I have self-hosted gitlab-ee instance with ultimate license.

I’ve been trying to connect gitlab-agent with gitlab for integration with k8s for a while, but i faced some issue:

First of all, i configured /etc/gitlab/gitlab.rb file with the following details:

gitlab_rails['gitlab_**kas**_enabled'] = true

gitlab_rails['gitlab_**kas**_external_url'] = 'wss://my-domain.com/-/kubernetes-agent/'

gitlab_rails['gitlab_**kas**_internal_url'] = 'grpc://localhost:8153'

gitlab_rails['gitlab_**kas**_external_k8s_proxy_url'] = 'https://my-domain.com/-/kubernetes-agent/'

gitlab_kas['enable'] = true

Next, i created new default cluster connection in infrastructure section in my project and got access token with helm installation instructions.

After that i’ve executed the following command on kubernetes control plane:

helm upgrade --install k8s-test gitlab/gitlab-agent \
     --namespace gitlab-agent \
     --create-namespace \
     --set image.tag=v15.6.0 \
     --set config.token=1jdQTxxxxgmeQ \
     --set config.kasAddress=wss://my-domain.com/-/kubernetes-agent/ --set config.caCert="$(cat /home/user/kube_config/agent/gitlabInstanceCert.crt)"

And finally, when i’ve seen the following logs in my pod, i don’t have connection established status on the gitlab side:

{“level”:“info”,“time”:“2022-12-20T20:12:06.068Z”,“msg”:“Observability endpoint is up”,“mod_name”:“observability”,“net_network │
│ “:“tcp”,“net_address”:”:8080”}

I was trying to troubleshoot and checked logs directly on my gitlab instance, i’ve seen the following there:

...
2022-12-20_20:29:20.31177 {"level":"error","time":"2022-12-20T20:29:20.311Z","msg":"AgentInfo()","grpc_service":"gitlab.agent.reverse_tunnel.rpc.ReverseTunnel","grpc_method":"Connect","error":"Get \"https://my-domain.com/api/v4/internal/kubernetes/agent_info\": x509: certificate signed by unknown authority"}
2022-12-20_20:30:30.52026 {"level":"error","time":"2022-12-20T20:30:30.520Z","msg":"AgentInfo()","grpc_service":"gitlab.agent.agent_configuration.rpc.AgentConfiguration","grpc_method":"GetConfiguration","error":"Get \"https://my-domain.com/api/v4/internal/kubernetes/agent_info\": x509: certificate signed by unknown authority"}
2022-12-20_20:31:55.08583 {"level":"error","time":"2022-12-20T20:31:55.085Z","msg":"AgentInfo()","grpc_service":"gitlab.agent.agent_configuration.rpc.AgentConfiguration","grpc_method":"GetConfiguration","error":"Get \"https://my-domain.com/api/v4/internal/kubernetes/agent_info\": x509: certificate signed by unknown authority"}
2022-12-20_20:32:09.43170 {"level":"error","time":"2022-12-20T20:32:09.431Z","msg":"AgentInfo()","grpc_service":"gitlab.agent.reverse_tunnel.rpc.ReverseTunnel","grpc_method":"Connect","error":"Get \"https://my-domain.com/api/v4/internal/kubernetes/agent_info\": x509: certificate signed by unknown authority"}
2022-12-20_20:33:26.57574 {"level":"error","time":"2022-12-20T20:33:26.575Z","msg":"AgentInfo()","grpc_service":"gitlab.agent.agent_configuration.rpc.AgentConfiguration","grpc_method":"GetConfiguration","error":"Get \"https://my-domain.com/api/v4/internal/kubernetes/agent_info\": x509: certificate signed by unknown authority"}

My gilab instance has letsencrypt certificate.

I was trying to play around with kas configuration in gitlab.rb but still no result. I see that gitlab instance not linked with my k8s cluster.

As shown in logs, gitlab instance and gitlab-agent in cluster are communicating with each other every 20-30 seconds.

Any thoughts about solution?

Gitlab version: 15.6

Appreciate any help or ideas.

1 Like
2

Same issue for me

3

i do have the exact same problem. I also tried to set the certificate of the gitlab-instance (even tough it’s a public paid certificate) and also i tried to set the self signed certificate of the Kubernetes API Server. Both with no success.

helm upgrade --install dev01 gitlab/gitlab-agent \
    --namespace gitlab-agent \
    --create-namespace \
    --set image.tag=v15.0.0 \
    --set config.token=<TOKEN> \
    --set config.kasAddress=wss://<DOMAIN>/-/kubernetes-agent/ \
    --set config.caCert="$(cat ./certificate.crt)"
4

I am also having this issue, if anyone knows how to resolve it I would be greatly appreciative.

5

SOLUTION:

I encountered this error too, you have to add your root self-signed certificate that gitlab certificate is signed with (or ssl, depending on how you did it) to the trusted-certs directory (/etc/gitlab/trusted-certs/)

As I understand gitlab kas service redirects requests not through localhost, but through external https interface of gitlab itself, because of which it may not know about it (especially if gitlab server is deployed in the docker).

To summarize:

  1. You have a certificate /opt/rootca.crt (rename it to rootca.pem).
  2. Add the certificate to gitlab (/etc/gitlab/trusted-certs)
cp /opt/rootca.pem /etc/gitlab/trusted-certs/
# restart gitlab or reconfigure
gitlab-ctl reconfigure
#or
docker restart gitlab
  1. Install helm agent
helm repo add gitlab https://charts.gitlab.io
helm repo update
helm upgrade --install my-agent gitlab/gitlab-agent \
    --namespace gitlab-agent \.
    --create-namespace.
    --set image.tag=v17.0.0.
    --set config.token=xxxxxxxxxxxxxxxxxxxxxxx
    --set config.kasAddress=wss://gitlab.my.domain/-/kubernetes-agent/\
    --set-file config.kasCaCert=/opt/rootca.pem
  1. Enjoy.