Gitaly Cluster - Praefect and HAProxy "6 hour" Timeout


I’m following the documentation instructions for setting up a gitlay cluster. I’m configuring the load balancer for Praefect using HAProxy, using a configuration similar to this one.

The load-balancer documentation contains this warning:

Long-running background jobs can maintain an idle connection with Praefect for up 6 hours. Set your load balancer timeout to be at least 6 hours long.

Ok. So any ideas about which of these timeout settings to use in HAProxy for this? I don’t think I want all of these set for 6 hours or this could lead to performance issues if a connection gets missed or something? Or perhaps I should just set all of them? These are the available timeouts with the default values below. Not sure which is best to use for this:

    timeout http-request    10s
    timeout queue           1m
    timeout connect         10s
    timeout client          1m
    timeout server          1m
    timeout http-keep-alive 10s
    timeout check           10s

I do have tcp keep-alive enabled (options srvtcpka and clitcpka), so possibly none of the other timeouts are even relevant? Not sure really.

In this scenario, it would seem the timeout server would be the relevant timeout:

In our GitLab-Environment-Toolkit “GET” we specify 30 minutes for server and client timeout as a baseline:

We later bumped that to 361 minutes (6 hours 1 minute)

Which we later reverted, because of the gitaly timeout change in release v14.5:

Basically, the optimal choice would be to move to v14.5 as that incorporates the gitaly timeout modifications. Second to that, setting 361 minutes for timeout server and timeout client would work.

1 Like