Gitlab Kas Kube Cluster Connection Woes

I have a Self hosted Gitlab omnibus install (quite old in terms of its been the same install continuously upgraded for about 6 years) and I can not get the kubernetes agent to connect back to it. Gitlab is behind an nginx reverse proxy (on a separate VM) that handles SSL termination. This then talks back to Gitlab via self signed SSL.

I have a the websocket section defined in my nginx config as such

proxy_http_version      1.1;
proxy_set_header        Upgrade $http_upgrade;
proxy_set_header        Connection "upgrade";
proxy_read_timeout      86400;

I can reach Gitlab KAS at the URL so I think its enabled correctly. I have tried both ws and wss in my callback agent command but both produce the same error. I have also tried going direct without the reverse proxy but get an SSL cert error

The error on the agent is

{"level":"error","time":"2022-01-15T14:17:52.349Z","msg":"Error handling a connection","mod_name":"reverse_tunnel","error":"Connect(): rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing failed to WebSocket dial: expected handshake response status code 101 but got 301\""}

Any ideas on troubleshooting this

To add some more info, the gitlab server and kubernetes cluster are hosted in different environments connected via a site-to-site vpn

If i browser to the URL for KAS i get the follow:

WebSocket protocol violation: Connection header "close" does not contain Upgrade

This error occurs if the kas-address doesn’t include a trailing slash. To fix it, make sure that the wss or ws URL ends with a trailing slash, such as wss://GitLab.host.tld:443/-/kubernetes-agent/ or ws://GitLab.host.tld:80/-/kubernetes-agent/ .

See GitLab Agent for Kubernetes | GitLab