Gitlab KAS. Problem with k8s connection

Hello there.

I have self-hosted gitlab-ee instance with ultimate license.

I’ve been trying to connect gitlab-agent with gitlab for integration with k8s for a while, but i faced some issue:

First of all, i configured /etc/gitlab/gitlab.rb file with the following details:

gitlab_rails['gitlab_**kas**_enabled'] = true

gitlab_rails['gitlab_**kas**_external_url'] = 'wss://my-domain.com/-/kubernetes-agent/'

gitlab_rails['gitlab_**kas**_internal_url'] = 'grpc://localhost:8153'

gitlab_rails['gitlab_**kas**_external_k8s_proxy_url'] = 'https://my-domain.com/-/kubernetes-agent/'

gitlab_kas['enable'] = true

Next, i created new default cluster connection in infrastructure section in my project and got access token with helm installation instructions.

After that i’ve executed the following command on kubernetes control plane:

helm upgrade --install k8s-test gitlab/gitlab-agent \
     --namespace gitlab-agent \
     --create-namespace \
     --set image.tag=v15.6.0 \
     --set config.token=1jdQTxxxxgmeQ \
     --set config.kasAddress=wss://my-domain.com/-/kubernetes-agent/ --set config.caCert="$(cat /home/user/kube_config/agent/gitlabInstanceCert.crt)"

And finally, when i’ve seen the following logs in my pod, i don’t have connection established status on the gitlab side:

{“level”:“info”,“time”:“2022-12-20T20:12:06.068Z”,“msg”:“Observability endpoint is up”,“mod_name”:“observability”,“net_network │
│ “:“tcp”,“net_address”:”:8080”}

I was trying to troubleshoot and checked logs directly on my gitlab instance, i’ve seen the following there:

...
2022-12-20_20:29:20.31177 {"level":"error","time":"2022-12-20T20:29:20.311Z","msg":"AgentInfo()","grpc_service":"gitlab.agent.reverse_tunnel.rpc.ReverseTunnel","grpc_method":"Connect","error":"Get \"https://my-domain.com/api/v4/internal/kubernetes/agent_info\": x509: certificate signed by unknown authority"}
2022-12-20_20:30:30.52026 {"level":"error","time":"2022-12-20T20:30:30.520Z","msg":"AgentInfo()","grpc_service":"gitlab.agent.agent_configuration.rpc.AgentConfiguration","grpc_method":"GetConfiguration","error":"Get \"https://my-domain.com/api/v4/internal/kubernetes/agent_info\": x509: certificate signed by unknown authority"}
2022-12-20_20:31:55.08583 {"level":"error","time":"2022-12-20T20:31:55.085Z","msg":"AgentInfo()","grpc_service":"gitlab.agent.agent_configuration.rpc.AgentConfiguration","grpc_method":"GetConfiguration","error":"Get \"https://my-domain.com/api/v4/internal/kubernetes/agent_info\": x509: certificate signed by unknown authority"}
2022-12-20_20:32:09.43170 {"level":"error","time":"2022-12-20T20:32:09.431Z","msg":"AgentInfo()","grpc_service":"gitlab.agent.reverse_tunnel.rpc.ReverseTunnel","grpc_method":"Connect","error":"Get \"https://my-domain.com/api/v4/internal/kubernetes/agent_info\": x509: certificate signed by unknown authority"}
2022-12-20_20:33:26.57574 {"level":"error","time":"2022-12-20T20:33:26.575Z","msg":"AgentInfo()","grpc_service":"gitlab.agent.agent_configuration.rpc.AgentConfiguration","grpc_method":"GetConfiguration","error":"Get \"https://my-domain.com/api/v4/internal/kubernetes/agent_info\": x509: certificate signed by unknown authority"}

My gilab instance has letsencrypt certificate.

I was trying to play around with kas configuration in gitlab.rb but still no result. I see that gitlab instance not linked with my k8s cluster.

As shown in logs, gitlab instance and gitlab-agent in cluster are communicating with each other every 20-30 seconds.

Any thoughts about solution?

Gitlab version: 15.6

Appreciate any help or ideas.

1 Like

Same issue for me

i do have the exact same problem. I also tried to set the certificate of the gitlab-instance (even tough it’s a public paid certificate) and also i tried to set the self signed certificate of the Kubernetes API Server. Both with no success.

helm upgrade --install dev01 gitlab/gitlab-agent \
    --namespace gitlab-agent \
    --create-namespace \
    --set image.tag=v15.0.0 \
    --set config.token=<TOKEN> \
    --set config.kasAddress=wss://<DOMAIN>/-/kubernetes-agent/ \
    --set config.caCert="$(cat ./certificate.crt)"

I am also having this issue, if anyone knows how to resolve it I would be greatly appreciative.