GitLab Agent for Kubernetes GRPC error

Hi,

We are using Gitlab Agent to deploy on our Kube clusters with the help of HELM.

We have this kind of error who are often popping in our CI/CD jobs :

Error: UPGRADE FAILED: release backend failed, and has been rolled back due to atomic being set: failed to refresh resource information: GitLab Agent Server: HTTP->gRPC: failed to read gRPC response: rpc error: code = Canceled desc = context canceled

Jobs are taking 3 to 5 minutes to deploy but this error is really random and is happening on multiple Kube clusters.

On the agent side we have errors who look like the one returned by the job

{“level”:“error”,“time”:“2022-11-22T10:04:13.396Z”,“msg”:“Error handling a connection”,“mod_name”:“reverse_tunnel”,“error”:“rpc error: code = Unavailable desc = error reading from server: failed to get reader: failed to read frame header: EOF”,“agent_id”:15764}

But they are never correlated to the ones we have in the CI.

It’s really annoying because when this is happening deployment is rollbacked and app can end in an inconsistent state.

Do you know how we can prevent this ?

Arnaud.

1 Like