Kubernetes agent fails to connect to GitLab with "expected handshake response status code 101 but got 426"

Infrastructure as Code & Cloud Native
1

We are successfully using a few kubernetes clusters integrated into self-hosted GitLab through the now-deprecated certificate-based connection.

I am in the process of setting up a new cluster and figured I would try out the recommended way. I want to use the CI/CD workflow. I followed the instructions:

  • Enabled the KAS (in Omnibus install)
  • Registered the agent as documented here
  • Created an empty configuration file for the agent
  • Installed the agent in the Kubernetes cluster (one-liner installation)

The agent is running but fails to do a handshake with the GitLab server. The logs just say this over and over:

{"level":"error","time":"2022-04-08T11:36:40.573Z","msg":"Error handling a connection","mod_name":"reverse_tunnel","error":"Connect(): rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing failed to WebSocket dial: expected handshake response status code 101 but got 426\""}

Looking at the GitLab server, there is a nginx reverse-proxy deployed in front of the Omnibus Docker image of GitLab. The nginx-in-omnibus access logs contains the following line:

172.18.0.1 - - [08/Apr/2022:12:08:15 +0000] "GET /-/kubernetes-agent/ HTTP/1.0" 426 81 "" "gitlab-agent/v14.10.0-rc1/f35edb0" -

Which means the request did get through to Omnibus fine. It then rejected the request with a 426 Upgrade Required (?).

I do not see any actual errors that would specify what is wrong and could not find any help with this issue online. The agent is registered in the project but its Connection status is Never connected .

From /var/log/gitlab/gitlab-kas/current:

2022-04-07_20:29:05.17900 {"level":"info","time":"2022-04-07T20:29:05.178Z","msg":"Kubernetes API endpoint is up","mod_name":"kubernetes_api","net_network":"tcp","net_address":"127.0.0.1:8154"}
2022-04-07_20:29:05.17935 {"level":"info","time":"2022-04-07T20:29:05.179Z","msg":"API endpoint is up","net_network":"tcp","net_address":"127.0.0.1:8153"}
2022-04-07_20:29:05.17952 {"level":"info","time":"2022-04-07T20:29:05.179Z","msg":"Private API endpoint is up","net_network":"tcp","net_address":"127.0.0.1:8155"}
2022-04-07_20:29:05.17987 {"level":"info","time":"2022-04-07T20:29:05.179Z","msg":"Agentk API endpoint is up","net_network":"tcp","net_address":"127.0.0.1:8150","is_websocket":true}
2022-04-07_20:29:05.17993 {"level":"info","time":"2022-04-07T20:29:05.179Z","msg":"Observability endpoint is up","mod_name":"observability","net_network":"tcp","net_address":"127.0.0.1:8151"}

Any ideas what could be wrong? Thanks!

2

I am running into the exact same issue. Gitlab is running behind an Apache reverse proxy which is also configured to proxy websocket requests. As you do, I can see in the gitlab logs, that requests are arriving at Gitlab but are responded with status code 426.

3

@melkamar @markushe
I had the same issue described here because my Nginx reverse proxy wasn’t passing along the Upgrade and Connection headers to GitLab, resulting in GitLab returning a 426. The Nginx docs talk about why this header isn’t passed along and how to do it.

I just added a new location block to my virtual host that adds the headers:

location /-/kubernetes-agent/ {
    proxy_set_header Host $host;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "upgrade";

    proxy_pass https://gitlab-server;
}
4

Thanks for the response. According to the following post and the Apache documentation these headers are already set automatically.

https://httpd.apache.org/docs/2.4/mod/mod_proxy_wstunnel.html

So for me there seems to be some other problem because that is exactly how our Apache revers proxy is configured (just changed the host and port here)

    ProxyPreserveHost On
    ProxyPass / http://myhost:1234/
    RewriteEngine on
    RewriteCond ${HTTP:Upgrade} websocket [NC]
    RewriteCond ${HTTP:Connection} upgrade [NC]
    RewriteRule /(.*) "ws://myhost:1234/$1" [P,L]

Anyone got this working behind an Apache proxy?

5

Works like a charm, love it. Thanks!

6

This is a bit old but I’m trying to get the new kubernetes agent running and it’s a real pain…
I use apache2 and had proxy issues as well (after wasting hours to understand that this is the issue).

Received 5xx errors, or also “yaml: line 3: mapping values are not allowed in this context”, 426 upgrade issues, etc.

I finally fixed the issue with the following proxy changes:

	# OR is required or agent won't connect
	RewriteCond %{HTTP:Upgrade} websocket [NC,OR]
	RewriteCond ${HTTP:Connection} upgrade [NC]
	RewriteCond %{REQUEST_URI} ^/-/kubernetes-agent/.* [NC]
	RewriteRule /(.*) ws://127.0.0.1:8150/$1 [P,L]

	# route proxy requests to 
	RewriteCond %{REQUEST_URI} ^/-/kubernetes-agent/k8s-proxy/.* [NC]
	RewriteRule /-/kubernetes-agent/k8s-proxy/(.*) http://127.0.0.1:8154/$1 [P,L]

Key issue was, that k8s-proxy must route to the kas api (kubernetes_api_listen_address), which is running on 8154 (default).

[Disclaimer: not an expert with apache2 proxy - just a happy guy who finally can deploy on kubernetes again :heart:]

1 Like