Slow git clones help

So I know this topic has been debugged a lot already. I think my issue is different. Hope some of you smart guys can give me ideas.

Here’s the numbers

• Full speed git clone is 50 mbps. I know this from cloning locally on our self-hosted GitLab server
• My computer clones at 7 mbps.
• If I boot my computer to safe mode, it clones at the full 50 mbps.
• I’ve tested cloning public repos on github, and same results: slow normally, but full speed in safe mode.

I’ve done loads of other tests, but I think the safe mode full speed clone rules out a LOT of potential issues, and proves it’s a software issue somewhere.

I’ve tried disabling windows firewall, antivirus, real-time protection, and any other malware or protection thing I could find on my machine, and it makes no difference at all. I’ve also tried cloning http vs https vs ssh, no difference.

Any other ideas?

The fact it works in safe mode, suggests that Windows is the problem here. Unfortunately, this is not a Windows forum. You would probably be best asking on the Microsoft Windows forums. This is a Gitlab forum, and cloning is not unique to Gitlab since they are generic git tools.

I’m not really sure how someone can help you here when this is a Windows problem? About the only thing I can suggest is make sure all your device drivers are up-to-date.

One answer from Windows forum: Redirecting

It’s possible windows is interfering due to a config somewhere, but the network is not slow. The link you sent me is for debugging slow networks. I can run speedtest.net to prove public network is fast. I’ve also ran iperf to do a bandwidth test between my computer and the GitLab server. It’s proven that I can transfer up to 800mbps between the machines.

Put another way, it’s really just git clones that are affected. If I go to a windows forum, they’ll just help me debug my network that’s not broken. Was hoping someone here had similar things they’ve tried disabling or configuring to help with slow git clones.

You mentioned that git clones work fine when Windows is booted in safe mode with network support? But when booted normally it’s not. That pretty much proves it’s a Windows problem, otherwise git would be slow when booted normally and when booted in safe mode.

Nobody has posted such questions previously, so potentially suggests problems with your WIndows installation. As mentioned, try updating all your drivers, run BIOS updates, ensure Windows is up-to-date, etc, etc. Also one more thing that came to mind, go into your network adaptor settings and disable power save - that I’ve seen before for network performance or rather lack of.

If you’re lucky and someone else has had this problem, maybe they will join in the conversation, but I expect from what you’ve written already, git isn’t the problem here - rather Windows unless it’s booted in safe mode, then it seems to work fine.

When things work in safe mode but not otherwise, it’s hard proof it’s a software problem, as opposed to a hardware problem. The point of the test rules out a lot of issues very fast. It’s more correct then just saying windows is broken. Quite possibly, there’s a windows config issue somewhere, but could just as easily be other software interference. Virus scanners, etc. Info on safe mode: Redirecting

An example of safe mode debugging: of one my recent new builds had awkward browser lag, ultimately related to a bug in some gaming optimization software pre installed with the PC. Proving it was a software problem saved eons of time. Still annoying though.

It’s also important data points that only got clones are affected. Iperf and other network tools, as well as regular downloads work perfect. Hence why it really doesn’t make sense to ask for help on a windows forum. A git forum maybe, I’m just used to gitlabs forums bc gitlab is awesome :).

Good news is my IT team has some extra old laptops i can play with and break. So thus is my opportunity to mess with and uninstall things till i find the culprit.

I disagree with this, unless you are including software problem as in device drivers for your network card. Since in safe mode, it could be using a different driver or with features disabled.

An example I experienced. WIFI speed working brilliant, stable, until NVidia drivers installed which then caused WIFI to be extremely slow, constant disconnects, etc, etc. I could boot safe mode, NVidia obviously would be disabled at this point, and then suddenly my network is high performance again.

You didn’t acknowledge whether you have attempted updating your drivers, system, BIOS from what I mentioned, or perhaps changing and disabling power management on the network connection? I’m guessing because you are already convinced that git is the problem here and not your system hence why not trying any of the suggestions so far? Maybe also ensure you are running the latest version of the git executable as well. However, I expect it is something on your system completely unrelated to git, and difficult to find or for anyone here to replicate.

Either way, good luck in finding your problem. Perhaps try a clean Windows installation or even clean Linux installation on the machine that has problems. If the problem persists in Windows, but doesn’t persist in Linux, that would suggest that Windows is the problem. Plenty of things to try to try and diagnose your problem at least.

Unless the old laptops are the same hardware wise as the machine you are having problems with, then I don’t see how that will help fix your problem. But at least you can try various OS’s on them without having to reinstall your problematic machine.

Phew finally got this one solved! Gonna post my debugging steps in case this helps someone. What I’ve found is there’s LOTS of slow git clone issues all over the place, and people have issues for all sorts of different reasons. Not always easy to debug.

The culprit

• Our company bought another company that had their own proprietary cyber scanning tools. We no longer use those tools, but they’re still installed on a lot of laptops. This tool was interfering with some software, like git, but not others, as per configuration. Simply had to uninstall, and solved everything!

Debugging steps

1. First, I tested local machine vs the server itself. So In otherwards, I ssh’d into the gitlab server and did a git clone while on the machine. This ruled out the server itself having issues.
2. Even with previous test, I went hunting through settings in gitlab. There’s a lot of throttles for things like API calls per user per minute. I did upgrade a few, but nothing really stuck out to me. Also installed cloudwatch since we’re on EC2 so I could watch RAM, CPU, network limits, etc, and the server was very underutilized.
3. Next ran all sorts of network tests to see if it was network problems in general, or more focused on git. There’s awesome tools like iperf3 that can test your bandwidth directly between two machines! Also downloaded a 7gb zip file from another one of our servers to further test network performance to our EC2s. All showed excellent network health! And made it clear the issue was more focused.
4. Next, did a fair bit of research on slow git clones. Various people have given suggestions to modify how caching is done, trying ssh vs https, etc. I tried those with no luck, though tbh I kinda doubt those kinds of things will fix something thats a core issue elsewhere, but if it works for someone, great!
   Next, tried disabling windows firewall, defender, etc, anything I could find that might scan or interfere with network traffic. Nothing seemed to make a difference.
5. Next, it was clear there was a deeper issue. Even if the great network health, wanted to rule out bad cables, firewall, switches, etc, anything that could be hardware, so booted up in safe mode+network option. This was my first successful test! And proof it was a software problem somewhere. For those who don’t know: in this context, software vs hardware issues means your problem may exist in windows itself, a driver somewhere, or other software services or apps running.
6. As a bonus test, I cloned a public repository that’s know to be massive. Thanks to reddit, I found this one: GitHub - NixOS/nixpkgs: Nix Packages collection & NixOS. It was the same issue where it was slow normally, but fast in safe mode. This helped verify there’s no difference of network traffic to our AWS region vs github.
7. Special thanks to my IT, was able to get a throw-away laptop. Verified same issue, and then got to work.
8. Booted the throw-away laptop in normal mode after initial tests, then started uninstalling and disabling things like crazy. Didn’t take long to find the culprit… An additional company scanning tool installed by a previous IT team was incorrectly interfering with git doing it’s thing.