Gitlab Kas Kube Cluster Connection Woes

GitLab CI/CD
1

I have a Self hosted Gitlab omnibus install (quite old in terms of its been the same install continuously upgraded for about 6 years) and I can not get the kubernetes agent to connect back to it. Gitlab is behind an nginx reverse proxy (on a separate VM) that handles SSL termination. This then talks back to Gitlab via self signed SSL.

I have a the websocket section defined in my nginx config as such

proxy_http_version      1.1;
proxy_set_header        Upgrade $http_upgrade;
proxy_set_header        Connection "upgrade";
proxy_read_timeout      86400;

I can reach Gitlab KAS at the URL so I think its enabled correctly. I have tried both ws and wss in my callback agent command but both produce the same error. I have also tried going direct without the reverse proxy but get an SSL cert error

The error on the agent is

{"level":"error","time":"2022-01-15T14:17:52.349Z","msg":"Error handling a connection","mod_name":"reverse_tunnel","error":"Connect(): rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing failed to WebSocket dial: expected handshake response status code 101 but got 301\""}

Any ideas on troubleshooting this

To add some more info, the gitlab server and kubernetes cluster are hosted in different environments connected via a site-to-site vpn

If i browser to the URL for KAS i get the follow:

WebSocket protocol violation: Connection header "close" does not contain Upgrade
2

This error occurs if the kas-address doesn’t include a trailing slash. To fix it, make sure that the wss or ws URL ends with a trailing slash, such as wss://GitLab.host.tld:443/-/kubernetes-agent/ or ws://GitLab.host.tld:80/-/kubernetes-agent/ .

See GitLab Agent for Kubernetes | GitLab

1 Like
3

How did you fixed the “WebSocket protocol violation: Connection header “close” does not contain Upgrade” ? I have all the trailing slashes i think.

gitlab_kas[‘enable’] = true
gitlab_kas[‘gitlab_address’] = ‘https://reg.domain.com’
gitlab_rails[‘gitlab_kas_enabled’] = true
gitlab_rails[‘gitlab_kas_external_url’] = ‘wss://reg.domain.com/-/kubernetes-agent/’
gitlab_rails[‘gitlab_kas_internal_url’] = ‘grpc://localhost:8153’
gitlab_rails[‘gitlab_kas_external_k8s_proxy_url’] = ‘https://reg.domain.com/-/kubernetes-agent/’

thanks

4

can anyone let me know how you guys created these secret keys.

gitlab_kas[‘api_secret_key’] = ‘<32_bytes_long_base64_encoded_value>’
gitlab_kas[‘private_api_secret_key’] = ‘<32_bytes_long_base64_encoded_value>’

@joes