Need help setting up gitlab-kas behind proxy

GitLab CI/CD
,
3

I was using the reference gitlab.rb template from https://github.com/gitlabhq/omnibus-gitlab/blob/master/files/gitlab-config-template/gitlab.rb.template#L1982 which had the gitlab_rails['gitlab_kas_enabled'] setting in it.

Anyway, switching from ws to wss for the external URL worked, but not without adding some websocket related settings to my reverse proxy.

I’ll just write out my solution for the sake of others.

Heres the working gitlab.rb settings

##! Settings used by the GitLab application
gitlab_rails['gitlab_kas_enabled'] = true
gitlab_rails['gitlab_kas_external_url'] = 'wss://gitlab.company.com/-/kubernetes-agent/'
gitlab_rails['gitlab_kas_internal_url'] = 'grpc://localhost:8153'

##! Define to enable GitLab KAS
gitlab_kas_external_url "wss://gitlab.company.com/-/kubernetes-agent/"
gitlab_kas['enable'] = true

Once I had set the external URL to wss I had to delete and re-install the helm release as it was still pointing to the ws address.

Now I was getting 426 errors instead of 301.

{"level":"error","time":"2023-09-21T06:52:31.417Z","msg":"Error handling a connection","mod_name":"reverse_tunnel","error":"Connect(): rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: failed to WebSocket dial: expected handshake response status code 101 but got 426\""}

When I investigated further, I found out that I needed to add some configurations to my nginx reverse proxy to support websockets.

Here is the nginx configuration for the reverse proxy I borrowed from u/ccoley Kubernetes agent fails to connect to GitLab with "expected handshake response status code 101 but got 426" - #3 by ccoley

server {
    server_name gitlab.company.com;

    location / {
        proxy_pass http://gitlab;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Real-IP $remote_addr;
    }

    location /-/kubernetes-agent/ {
        proxy_pass http://gitlab;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Real-IP $remote_addr;

        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
    }
}

It is important to note you need the trailing slash in the location for the kubernetes agent. I spent far too long trying to figure out why the proxy wasn’t working due to the missing slash.

1 Like