GitLab-Kas not working after upgrade

GitLab CI/CD
1

Our kubernetes configuration has worked fine for years and suddenly, after upgrading to 17.3.2, our agent on our k8 can’t connect to gitlab.
We are running GitLab 17.3.2-ee Omnibus. I don’t have any entries in gitlab.rb relating to gitlab-kas but I do have /var/opt/gitlab/gitlab-kas/gitlab-kas-config.yml which has the following:

agent:
listen:
address: localhost:8150
network: tcp
websocket: true
configuration:
poll_period: 300s
gitops:
poll_period: 300s
project_info_cache_ttl: 300s
project_info_cache_error_ttl: 60s
kubernetes_api:
listen:
address: localhost:8154
url_path_prefix: /
info_cache_ttl: 300s
info_cache_error_ttl: 60s
gitlab:
address: https://git..com
external_url: https://git..com
authentication_secret_file: /var/opt/gitlab/gitlab-kas/authentication_secret_file
observability:
listen:
address: localhost:8151
network: tcp
usage_reporting_period: 60s
logging:
level: info
grpc_level: error
redis:
network: unix
tls:
enabled: false
server:
address: ‘/var/opt/gitlab/redis/redis.socket’
api:
listen:
network: tcp
address: localhost:8153
authentication_secret_file: /var/opt/gitlab/gitlab-kas/authentication_secret_file
private_api:
listen:
network: tcp
address: localhost:8155
authentication_secret_file: /var/opt/gitlab/gitlab-kas/private_api_authentication_secret_file

The only thing that has changed is upgrading to 17.3.2 and running a reconfigure.
gitlab-status shows gitlab-kas as up
Our k8 cluster has the latest 17.3.2 agent running.
I am not running this behind a proxy.

The error from the agent on our cluster is

{“time”:“2024-09-16T12:06:32.836734408Z”,“level”:“ERROR”,“msg”:“Failed to register agent pod. Please make sure the agent version matches the server version”,“mod_name”:“agent_registrar”,“error”:“rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: failed to WebSocket dial: expected handshake response status code 101 but got 400"”}

2

I believe you should have configured in /etc/gitlab/gitlab.rb, see below:

################################################################################
## GitLab Kubernetes Agent Server
##! Docs: https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent/blob/master/README.md
################################################################################

##! Settings used by the GitLab application
# gitlab_rails['gitlab_kas_enabled'] = true
# gitlab_rails['gitlab_kas_external_url'] = 'ws://gitlab.example.com/-/kubernetes-agent/'
# gitlab_rails['gitlab_kas_internal_url'] = 'grpc://localhost:8153'
# gitlab_rails['gitlab_kas_external_k8s_proxy_url'] = 'https://gitlab.example.com/-/kubernetes-agent/k8s-proxy/'

##! Define to enable GitLab KAS
# gitlab_kas_external_url "ws://gitlab.example.com/-/kubernetes-agent/"
# gitlab_kas['enable'] = false

##! Agent configuration for GitLab KAS
# gitlab_kas['agent_configuration_poll_period'] = 300
# gitlab_kas['agent_gitops_poll_period'] = 300
# gitlab_kas['agent_gitops_project_info_cache_ttl'] = 300
# gitlab_kas['agent_gitops_project_info_cache_error_ttl'] = 60
# gitlab_kas['agent_info_cache_ttl'] = 300
# gitlab_kas['agent_info_cache_error_ttl'] = 60

##! Shared secret used for authentication between KAS and GitLab
# gitlab_kas['api_secret_key'] = nil # Will be generated if not set. Base64 encoded and exactly 32 bytes long.

##! Shared secret used for authentication between different KAS instances in a multi-node setup
# gitlab_kas['private_api_secret_key'] = nil # Will be generated if not set. Base64 encoded and exactly 32 bytes long.

##! Listen configuration for GitLab KAS
# gitlab_kas['listen_address'] = 'localhost:8150'
# gitlab_kas['listen_network'] = 'tcp'
# gitlab_kas['listen_websocket'] = true
# gitlab_kas['certificate_file'] = "/path/to/certificate.pem"
# gitlab_kas['key_file'] = "/path/to/key.pem"
# gitlab_kas['observability_listen_network'] = 'tcp'
# gitlab_kas['observability_listen_address'] = 'localhost:8151'
# gitlab_kas['internal_api_listen_network'] = 'tcp'
# gitlab_kas['internal_api_listen_address'] = 'localhost:8153'
# gitlab_kas['internal_api_certificate_file'] = "/path/to/certificate.pem"
# gitlab_kas['internal_api_key_file'] = "/path/to/key.pem"
# gitlab_kas['kubernetes_api_listen_address'] = 'localhost:8154'
# gitlab_kas['kubernetes_api_certificate_file'] = "/path/to/certificate.pem"
# gitlab_kas['kubernetes_api_key_file'] = "/path/to/key.pem"
# gitlab_kas['private_api_listen_network'] = 'tcp'
# gitlab_kas['private_api_listen_address'] = 'localhost:8155'
# gitlab_kas['private_api_certificate_file'] = "/path/to/certificate.pem"
# gitlab_kas['private_api_key_file'] = "/path/to/key.pem"

##! Metrics configuration for GitLab KAS
# gitlab_kas['metrics_usage_reporting_period'] = 60

##! Log configuration for GitLab KAS
# gitlab_kas['log_level'] = 'info'

##! Environment variables for GitLab KAS
# gitlab_kas['env'] = {
#   'SSL_CERT_DIR' => "/opt/gitlab/embedded/ssl/certs/",
#   # In a multi-node setup, this address MUST be reachable from other KAS instances. In a single-node setup,
#   # it can be on localhost for simplicity.
#   # Use OWN_PRIVATE_API_CIDR + OWN_PRIVATE_API_PORT (optional) + OWN_PRIVATE_API_SCHEME (optional) if you cannot
#   # specify a correct address for each KAS instance in OWN_PRIVATE_API_URL.
#   'OWN_PRIVATE_API_URL' => 'grpc://localhost:8155'
#   # 'OWN_PRIVATE_API_CIDR' => '10.0.0.0/8', # IPv4 example
#   # 'OWN_PRIVATE_API_CIDR' => '2001:db8:8a2e:370::7334/64', # IPv6 example
#   # 'OWN_PRIVATE_API_PORT' => '8155', # if not set, port from private_api_listen_address is used
#   # 'OWN_PRIVATE_API_SCHEME' => 'grpc', # use grpcs when using TLS on private API endpoint
# }

##! Error Reporting and Logging with Sentry
# gitlab_kas['sentry_dsn'] = 'https://<key>@sentry.io/<project>'
# gitlab_kas['sentry_environment'] = 'production'

##! Directories for GitLab KAS
# gitlab_kas['dir'] = '/var/opt/gitlab/gitlab-kas'
# gitlab_kas['log_directory'] = '/var/log/gitlab/gitlab-kas'
# gitlab_kas['log_group'] = nil
# gitlab_kas['env_directory'] = '/opt/gitlab/etc/gitlab-kas/env'

##! Redis settings for GitLab KAS
# gitlab_kas['redis_socket'] = ''
# gitlab_kas['redis_host'] = '127.0.0.1'
# gitlab_kas['redis_port'] = '6379'
# gitlab_kas['redis_password'] = nil

# gitlab_kas['redis_sentinels'] = []
# gitlab_kas['redis_sentinels_master_name'] = nil
# gitlab_kas['redis_sentinels_password'] = ''

# gitlab_kas['redis_ssl'] = false
# gitlab_kas['redis_tls_ca_cert_file'] = '/opt/gitlab/embedded/ssl/certs/cacert.pem'
# gitlab_kas['redis_tls_client_cert_file'] = nil
# gitlab_kas['redis_tls_client_key_file'] = nil

##! Command to generate extra configuration
# gitlab_kas['extra_config_command'] = nil

those are all the config options available in /etc/gitlab/gitlab.rb that you should most likely be configuring if you wish to use kas.