Error when trying to connect k8s cluster with Gitlab

Infrastructure as Code & Cloud Native
1

Hi all,
I have a Omnibus Gitlab CE 14.10.5 running with docker compose

version: '3.6'
services:
  gitlab:
    image: 'gitlab/gitlab-ce:14.10.5-ce.0'
    restart: always
    hostname: 'gitlab.mydomain.com'
    container_name: gitlab
    environment:
      - TZ=Asia/Ho_Chi_Minh
    ports:
      - '80:80' # my gitlab run http only
      - '22:22'
    volumes:
      - /srv/gitlab/config:/etc/gitlab
      - /srv/gitlab/logs:/var/log/gitlab
      - /srv/gitlab/data:/var/opt/gitlab
      - /srv/gitlab/backup:/var/opt/gitlab/backups
    shm_size: '512m'

It works fine, I edit gitlab.rb

gitlab_kas['enable'] = true
gitlab_kas['gitlab_address'] = 'http://gitlab.mydomain.com'

then reconfigure gitlab.
On my Gitlab I create a project " setup_gitlab_agent" with a file .gitlab/agents/testagent/config.yaml

observability:
    logging:
        level: debug

Go to Infrastructure → Kubernetes clusters → Agent → Connect a cluster (agent) → choose “testagent” above .
I connect to my k8s cluster on premise and install Gitlab agent

kubectl create ns gitlab-agent
helm upgrade --install gitlab-agent gitlab/gitlab-agent \
    --namespace gitlab-agent \
    --set config.token=******** \
    --set config.kasAddress=ws://gitlab.mydomain.com/-/kubernetes-agent \
    --set image.tag=v14.10.0/

I edit /etc/hosts so that gitlab.mydomain.com can be resolved on my Gitlab server , k8s nodes
But gitlab-agent pod shows error
{"level":"error","time":"2023-03-28T04:20:26.340Z","msg":"Error handling a connection","mod_name":"reverse_tunnel","error":"Connect(): rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing failed to WebSocket dial: failed to send handshake request: Get \\\"http://gitlab.mydomain.com/-/kubernetes-agent\\\": dial tcp: lookup gitlab.mydomain.com on 10.96.0.10:53: no such host\""}
How can I troubleshoot and make it works ? How to make sure connectivity between the kas-address and gitlab agent pod is ok ?

2

I fixed it.
Somehow /etc/hosts file on my k8s nodes and Gitlab server doesn’t work , so gitlab agent pod cannot resolve gitlab.mydomain.com , after that I get error

"error": "Connect(): rpc error: code = Unavailable desc = connection error: desc= \"transport: Error while dialing failed to WebSocket dial: expected handshake response status code 101 but got 301\""

But I fixed it by upgrade gitlab-agent release by helm , add a trailing slash to URL
--set config.kasAddress=ws://gitlab.mydomain.com/-/kubernetes-agent/