Docker container cloning large repo fails (solved)

I have a server running GitLab CE in a docker container using gitlab/gitlab-ce:latest and GitLab docker runner using gitlab/gitlab-runner:latest. The GitLab server also sits behind and nginx reverse proxy that terminates ssl using a let’s encrypt certificate.

The problem seems to be a combination of a large git repository (>~100MB), cloning via https (which gitlab-runner does), cloning within docker container (my gitlab-runner is). The error is also probabilistic happens about 80%-90% of the time. The following is the output of an attempted clone that fails:

error: RPC failed; result=56, HTTP code = 200
fatal: The remote end hung up unexpectedly
fatal: early EOF
fatal: index-pack failed

This seems to be a “common” issue with git as a client:

I tried many things:

  • double checking nginx timeouts
  • allowed cipher suites for ssl
  • max body size
  • reverse proxy settings
  • recompiling git client with openssl instead of gnutls
  • docker mtu size

I finally came across a post about docker network and having “similar” issues:

On my server I set

echo 1 > /proc/sys/net/netfilter/nf_conntrack_tcp_be_liberal

And haven’t had any issue cloning in my docker containers and GitLab runner over https.