Gitlab KAS Agent connection issue

Infrastructure as Code & Cloud Native
, ,
1

we configured gitlab-agent with eks cluster to execute runners on this same eks cluster . we have followed below helm commands and link.
->helm repo add gitlab https://charts.gitlab.io
->helm repo update
->helm upgrade --install gitlab-agent gitlab/gitlab-agent
–namespace gitlab-agent
–set config.token=XXXXXXXXXXXXXXXXXX
–set config.kasAddress=wss://XXXXXXXX-/kubernetes-agent/

we can see the agent status is connected in UI

image
image1505×193 12.9 KB

But when we tried to execute the pipeline we are getting Below error

$ kubectl get pods -n gitlab-agent-esp

[50](https://xxxx/repos/testrepoinvite/test1/-/jobs/8945281#L50)Unable to connect to the server: net/http: request canceled (Client.Timeout exceeded while awaiting headers)

below is the config.yaml

/repos/testrepoinvite/.gitlab/agents/gitlab-agent/config.yaml

ci_access:
  groups:
    - id: repos/testrepoinvite
observability:
  logging:
    level: debug
    grpc_level: error

Below is the gitlab-ci.yaml config file:

repos/testrepoinvite/gitlab-ci.yaml

deploy:
  tags:
    - prod-test-runner
  image:
    name: bitnami/kubectl:latest
    entrypoint: ['']
  script:
    - kubectl config get-contexts
    - kubectl config use-context repos/testwrepoinvite/test1:gitlab-agent
    - kubectl config view
    #- kubectl cluster-info
    - kubectl get pods -n gitlab-agent-esp

Below is the logs output from tail -f /var/log/gitlab/gitlab-kas/current this location

2022-07-05_04:52:36.23625 {"level":"error","time":"2022-07-05T04:52:36.236Z","msg":"RecvMsg(GatewayKasResponse)","correlation_id":"01G769S1CPDZF5SYRAAZF94RZQ","grpc_service":"gitlab.agent.kubernetes_api.rpc.KubernetesApi","grpc_method":"MakeRequest","agent_id":22,"kas_url":"grpc://localhost:8155","error":"rpc error: code = Unavailable desc = closing transport due to: connection error: desc = \"error reading from server: EOF\", received prior goaway: code: NO_ERROR"}

2022-07-05_04:58:58.95746 {"level":"info","time":"2022-07-05T04:58:58.957Z","msg":"Config: new commit","correlation_id":"01G76BKMDFQBS8WC72FX9D7HE7","grpc_service":"gitlab.agent.agent_configuration.rpc.AgentConfiguration","grpc_method":"GetConfiguration","agent_id":22,"project_id":"repos/testrepoinvite/test1","commit_id":"f8a4945de9696771e99973bea143fa787965c95c"}

2022-07-05_04:58:58.99821 {"level":"info","time":"2022-07-05T04:58:58.998Z","msg":"Config: failed to fetch","correlation_id":"01G76BKMDFQBS8WC72FX9D7HE7","grpc_service":"gitlab.agent.agent_configuration.rpc.AgentConfiguration","grpc_method":"GetConfiguration","agent_id":22,"project_id":"repos/testrepoinvite/test1","error":"failed to parse agent configuration: protojson.Unmarshal: proto: (line 1:89): unknown field \"grpc_level\""}

gitlab version we are using is 15.0.4 community edition
kubernetes version
Client Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.0", GitCom mit:"4ce5a8954017644c5420bae81d72b09b735c21f0", GitTreeState:"clean", BuildDate: "2022-05-04T02:28:17Z", GoVersion:"go1.18.1", Compiler:"gc", Platform:"linux/amd 64"}

In gitlab.rb file we included this line
gitlab_kas['enable'] = true

can someone please help us on this

2

failed to parse agent configuration: protojson.Unmarshal: proto: (line 1:89): unknown field \"grpc_level\""}

The grpc_level setting was added in GitLab 15.1 and as such, the agent with 15.0.4 does not know about it, erroring out. Suggest removing the setting until you find time to upgrade the agent deployment to 15.1.

3

Hi @dnsmichi We removed the grpc_level setting and retriggered the pipeline, but we are getting the below error in the pipeline

$ kubectl get pods -n gitlab-agent-esp

[50](https://xxxx/repos/testrepoinvite/test1/-/jobs/8945281#L50)Unable to connect to the server: net/http: request canceled (Client.Timeout exceeded while awaiting headers)

Below is the logs output from tail -f /var/log/gitlab/gitlab-kas/current this location

2022-07-06_08:37:53.05993 {"level":"info","time":"2022-07-06T08:37:53.059Z","msg":"HTTP->gRPC: failed to write HTTP response","mod_name":"kubernetes_api","correlation_id":"01G79A4E3BMENC0Z7K88XFMF4G","agent_id":22,"error":"write tcp 127.0.0.1:8154->127.0.0.1:45940: i/o timeout"}

can you please help us, solve this issue

4

Hi @andedhanesh same way we have configured agent. in your case agent is in connected state. in my case it is not in connected state. can you please review my issue link below and give some needful suggestions.

5

Hi @maheshmalidevi001 In your gitlab.rb file did you enable this

##### Installing K8 agents : https://docs.gitlab.com/ee/user/clusters/agent/install/    #####
gitlab_kas['enable'] = true
6

Hi @dnsmichi We were able to finally analyze the issue that we were having with the Kubernetes Agent Server, we saw that if we had only one instance of GitLab running the Kubernetes Agent Server works without any issues but in our case we have a multi-node setup for GitLab and the KAS server is not working.

I was able to find a couple of issues talking about this use case but I couldn’t get a straight forward way of implementing the changes suggested

[gitlab-kas enabled by default from 14.8 - including to multiple server types in scaled out deployments (#354304) · Issues · GitLab.org / GitLab · GitLab]

[Update reference architecture docs with KAS examples (#354573) · Issues · GitLab.org / GitLab · GitLab]

Do you have any documentation that can guide me in setting up the KAS for this use case or some example gitlab.rb templates?

Thanks
Dhanesh