Kubernetes runner fails sometime

GitLab CI/CD
1

Hi,

I got a self-hosted gitlab deployed in EKS using Classic ELB (port 22 and 443 enabled).
I have 2 running runners.

  • one is K8s runner which will do the deployment to prod namespace.
  • the other one is EC2 runner which provisions docker-machine runner for testing.

Every function works fine, except the K8s runner sometime failed to fetch one repository with below error

error: RPC failed; curl 18 transfer closed with outstanding read data remaining
error: 3872 bytes of body are still expected
02:18:22.362421 pkt-line.c:80           packet:          git> 0002
fetch-pack: unexpected disconnect while reading sideband packet
fatal: early EOF
fatal: fetch-pack: invalid index-pack output

This repo isn’t too large (1.34 GiB), even smaller than the others in our system. We have to desperately retry till it success.

Screen Shot 2024-03-07 at 15.21.54
Screen Shot 2024-03-07 at 15.21.54534×646 81.8 KB

What i have tried so far:

  • Update gitlab instance & runner to v16.7.4
  • Set get_sources_attempts to 5
  • Set job log level at debug
  • Set runner mem limit to 2GB
  • Set http.postBuffer 524288000

Has anyone face this problem before ? I’m not sure what should i try next, may switch to clone via SSH ?

Any suggestion would be highly appreciated !

2

Hello,
@vinhoe Did you manage to solve this issue please?

Thanks for your feedback.

3

@oelouafi Thanks for asking, unfortunately not. But i think the root cause was gitaly
I was testing with same runner:

  • if i run with git_strategy = none, it’s working fine all the time
  • if the runner clones the repo, it might fails.

The gitaly container seems not responding to request from this kubernetes runner, but the docker-machine runner is not affected.
I have updated those gitaly variables recently. Also because i use ingress nginx, i have extended the proxy-connect-timeout annotation to 60s
For the last 2 weeks, the retry times has decreased, which is nice, but i’m 100% sure this is not the solution because sometimes it’s still need to attempt twice to connect to that repository.
Please try increasing the gitaly timeout and let me know if it helps.

4

@vinhoe thanks for the update.
I solved the issue by adding this variable into my gitlab-ci file:

variables:
GIT_HTTP_POST_BUFFER: 157286400

it’s the same as applying this config:

git config --global http.postBuffer 157286400

1 Like