Slow "git fetch" and "pull" commands after upgrade to 14.8.0

Hi,

Have just finished an upgrade from version 14.5.2 to 14.8.0 and most things are working well. However, we are seeing users complaining (and have confirmed that it is the case) that commands such as:

git fetch -pa
git pull

are taking ~25-30seconds when prior to the upgrade they were taking ~1-2seconds.

We have confirmed that this appears to only be affecting git+SSH (running commands over HTTPS is still fast).

I noticed that there is a patch release out GitLab Patch Release: 14.8.1 | GitLab however, there is no mention of any fixes to issues related to SSH.

I do note that there were changes made to support new types of SSH keys in the 14.8.0 release. Could these be affecting performance?

Are there any other users experiencing similar issues?

What is the best way to go about diagnosing slow git+ssh connections in GitLab?

Ok, this turned out to be a systems issue, unrelated to the GitLab upgrade.

Specifically, it looks like the “systemd-logind” got into a bad state where it was unable to terminate user sessions, so possibly related to one of these bugs/issues:

If you are seeing similar behaviour, you should be able to diagnose whether this is the issue by running “loginctl list-sessions”, seeing a large number of sessions from the “git” user (or whatever user GitLab SSH runs as) for a start. Running “systemctl status systemd-logind” should also show the process in a restart loop (failing Watchdog/Health checks and being restarted constantly) as well as errors being thrown when attempting to terminate sessions manually with “loginctl terminate-session”.

Hope this can help someone’s day :slight_smile:

2 Likes