Kubernetes runner fails sometime


I got a self-hosted gitlab deployed in EKS using Classic ELB (port 22 and 443 enabled).
I have 2 running runners.

  • one is K8s runner which will do the deployment to prod namespace.
  • the other one is EC2 runner which provisions docker-machine runner for testing.

Every function works fine, except the K8s runner sometime failed to fetch one repository with below error

error: RPC failed; curl 18 transfer closed with outstanding read data remaining
error: 3872 bytes of body are still expected
02:18:22.362421 pkt-line.c:80           packet:          git> 0002
fetch-pack: unexpected disconnect while reading sideband packet
fatal: early EOF
fatal: fetch-pack: invalid index-pack output

This repo isn’t too large (1.34 GiB), even smaller than the others in our system. We have to desperately retry till it success.

What i have tried so far:

  • Update gitlab instance & runner to v16.7.4
  • Set get_sources_attempts to 5
  • Set job log level at debug
  • Set runner mem limit to 2GB
  • Set http.postBuffer 524288000

Has anyone face this problem before ? I’m not sure what should i try next, may switch to clone via SSH ?

Any suggestion would be highly appreciated !