Gitlab KAS connection issue

Infrastructure as Code & Cloud Native
1

Gitlab kubernetes agent is not establishing connection with local gitlab. When I checked the logs only log I see is

kubectl logs -f gitlab-agent-547b7984cd-r4q5q -n gitlab-kubernetes-agent
{“level”:“info”,“time”:“2022-01-21T13:47:32.279Z”,“msg”:“Feature status change”,“feature_name”:“tunnel”,“feature_status”:true}
{“level”:“info”,“time”:“2022-01-21T13:47:32.279Z”,“msg”:“Observability endpoint is up”,“mod_name”:“observability”,“net_network”:“tcp”,“net_address”:"[::]:8080"}

thats it. no other logs are getting emitted

this is agent deployment arg

  • args:
    - --token-file=/config/token
    - --kas-address
    - wss://gitlab.redacted.com:443/-/kubernetes-agent/

gitlab config for kas:
Enable GitLab KAS
gitlab_kas[‘enable’] = true

##! Listen configuration for GitLab KAS
gitlab_kas[‘listen_address’] = ‘0.0.0.0:8150’

2 Likes
2

Did you ever fix this? I found this to be frustratingly impossible to debug.

1 Like
3

I had this issue as well.

In my case I was running GitLab Omnibus on my own server, using docker and running behind a Traefik reverse proxy.

According to the troubleshooting information here:

Right at the bottom it speaks of checking the gitlab-kas logs (<GITLAB_ROOT>/logs/gitlab-kas/current) and about this message showing up:

{"level":"error","time":"2021-08-16T14:56:47.289Z","msg":"GetAgentInfo()","correlation_id":"01FD7QE35RXXXX8R47WZFBAXTN","grpc_service":"gitlab.agent.reverse_tunnel.rpc.ReverseTunnel","grpc_method":"Connect","error":"Get \"https://gitlab.example.com/api/v4/internal/kubernetes/agent_info\": dial tcp 172.17.0.4:443: connect: connection refused"}

Which I had, and seems to be because of the reverse proxy terminating the SSL certificate.

The resolution was to add this option to the /etc/gitlab/gitlab.rb file with your domain:

gitlab_kas['gitlab_address'] = 'http://gitlab.example.com'

Note that there is no S in http, this is what fixed it for me

1 Like
5

You are my hero!!
Btw somewhere i read that because websocket is connects client with server directly wss can’t be terminated and switched to ws. So it must be wss or ws all the way. Seems you found what needs to be fixed!

6

May somehting like this can help us?
proxy_set_header X-Forwarded-Port 443;
proxy_set_header X-Forwarded-Proto https;
proxy_set_header X-Forwarded-Ssl on;

if we place it in "location /-/kubernetes-agent/ " ?

7

Possibly, I haven’t tried anything more since the fix above solved my issues, and haven’t had any issues with the agent since.