Need help setting up gitlab-kas behind proxy

GitLab CI/CD
,
1

Hello!

I am trying to setup integration of an on-premise gitlab-ce omnibus deployment with a microk8s cluster but appear to have hit a small roadblock. Would appreciate any help at all!

Heres some context about my environment:
The GitLab service is running behind a reverse proxy that handles HTTPS.

[external traffic] - https -> [reverse proxy] - http -> [GitLab] <- ? -> [microk8s]

My initial gitlab-rb included the gitlab-kas section that looked like the following (actual hostname changed to “company.com”)

##! Settings used by the GitLab application
gitlab_rails['gitlab_kas_enabled'] = true
gitlab_rails['gitlab_kas_external_url'] = 'ws://gitlab.company.com/-/kubernetes-agent/'
gitlab_rails['gitlab_kas_internal_url'] = 'grpc://localhost:8153'
gitlab_rails['gitlab_kas_external_k8s_proxy_url'] = 'https://gitlab.company.com/-/kubernetes-agent/k8s-proxy/'

##! Define to enable GitLab KAS
gitlab_kas_external_url "ws://gitlab.company.com/-/kubernetes-agent/"
gitlab_kas['enable'] = true

Next I attempted to register an agent on my microk8s cluster by running the following commands

microk8s helm repo add gitlab https://charts.gitlab.io
microk8s helm repo update
microk8s helm upgrade --install k8s-integration-agent gitlab/gitlab-agent     --namespace gitlab-agent-k8s-integration-agent     --create-namespace     --set image.tag=v16.3.0     --set config.token=<redacted>     --set config.kasAddress=ws://gitlab.company.com/-/kubernetes-agent/

Things weren’t working right and I tried to look at some logs from the gitlab-agent pod.

{
	"level": "error",
	"time": "2023-09-11T06:29:51.035Z",
	"msg": "Error handling a connection",
	"mod_name": "reverse_tunnel",
	"error": "Connect(): rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: failed to WebSocket dial: expected handshake response status code 101 but got 301\""
}

After reading the troubleshooting guide at https://docs.gitlab.com/ee/user/clusters/agent/troubleshooting.html#transport-error-while-dialing-failed-to-websocket-dial, I double checked the addresses and found no problems with trailing slashes in my gitlab.rb file.

At this point I suspected that the issue was because I was running GitLab behind a proxy which helpfully redirects non-https to https.

I then tried to follow the instructions at https://docs.gitlab.com/ee/administration/clusters/kas.html#configure-kas-to-listen-on-a-unix-socket to use a unix socket but got errors on the cluster registration page.

image
image1533Ă—195 9.94 KB

I think its because I botched the gitlab.rb file which now looks like the following:

##! Settings used by the GitLab application
gitlab_rails['gitlab_kas_enabled'] = true
gitlab_rails['gitlab_kas_external_url'] = 'ws://gitlab.company.com/-/kubernetes-agent/'
#gitlab_rails['gitlab_kas_internal_url'] = 'grpc://localhost:8153'
gitlab_rails['gitlab_kas_internal_url'] = 'unix:///var/opt/gitlab/gitlab-kas/sockets/internal-api.socket'
gitlab_rails['gitlab_kas_external_k8s_proxy_url'] = 'https://gitlab.company.com/-/kubernetes-agent/k8s-proxy/'

##! Define to enable GitLab KAS
gitlab_kas_external_url "ws://gitlab.company.com/-/kubernetes-agent/"
gitlab_kas['enable'] = true

gitlab_kas['internal_api_listen_network'] = 'unix'
gitlab_kas['internal_api_listen_address'] = '/var/opt/gitlab/gitlab-kas/sockets/internal-api.socket'
gitlab_kas['private_api_listen_network'] = 'unix'
gitlab_kas['private_api_listen_address'] = '/var/opt/gitlab/gitlab-kas/sockets/private-api.socket'

##! Environment variables for GitLab KAS
gitlab_kas['env'] = {
  'SSL_CERT_DIR' => "/opt/gitlab/embedded/ssl/certs/",
  'OWN_PRIVATE_API_URL' => 'unix:///var/opt/gitlab/gitlab-kas/sockets/private-api.socket'
}

Did I mess up the gitlab_rails['gitlab_kas_internal_url'] setting somehow?

When attempting to register a new agent I get an Failed to register an agent GRPC::Unavailable error

2

Have you tried using wss (WebSocker over TLS) in stead of ws in external_urls?
Can you check that the socket files actually exists?

Try to simplify your gitlab.rb and remove what doesn’t needs to be there.
gitlab_rails['gitlab_kas_external_url'] by default equals to gitlab_kas_external_url, no need to specify again.
gitlab_rails['gitlab_kas_external_k8s_proxy_url'] is also generated from gitlab_kas_external_url, no need to specify.
gitlab_rails['gitlab_kas_enabled'] = true is really needed? Not specified anywhere in the install manual.

3

I was using the reference gitlab.rb template from https://github.com/gitlabhq/omnibus-gitlab/blob/master/files/gitlab-config-template/gitlab.rb.template#L1982 which had the gitlab_rails['gitlab_kas_enabled'] setting in it.

Anyway, switching from ws to wss for the external URL worked, but not without adding some websocket related settings to my reverse proxy.

I’ll just write out my solution for the sake of others.

Heres the working gitlab.rb settings

##! Settings used by the GitLab application
gitlab_rails['gitlab_kas_enabled'] = true
gitlab_rails['gitlab_kas_external_url'] = 'wss://gitlab.company.com/-/kubernetes-agent/'
gitlab_rails['gitlab_kas_internal_url'] = 'grpc://localhost:8153'

##! Define to enable GitLab KAS
gitlab_kas_external_url "wss://gitlab.company.com/-/kubernetes-agent/"
gitlab_kas['enable'] = true

Once I had set the external URL to wss I had to delete and re-install the helm release as it was still pointing to the ws address.

Now I was getting 426 errors instead of 301.

{"level":"error","time":"2023-09-21T06:52:31.417Z","msg":"Error handling a connection","mod_name":"reverse_tunnel","error":"Connect(): rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: failed to WebSocket dial: expected handshake response status code 101 but got 426\""}

When I investigated further, I found out that I needed to add some configurations to my nginx reverse proxy to support websockets.

Here is the nginx configuration for the reverse proxy I borrowed from u/ccoley Kubernetes agent fails to connect to GitLab with "expected handshake response status code 101 but got 426" - #3 by ccoley

server {
    server_name gitlab.company.com;

    location / {
        proxy_pass http://gitlab;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Real-IP $remote_addr;
    }

    location /-/kubernetes-agent/ {
        proxy_pass http://gitlab;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Real-IP $remote_addr;

        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
    }
}

It is important to note you need the trailing slash in the location for the kubernetes agent. I spent far too long trying to figure out why the proxy wasn’t working due to the missing slash.

1 Like