Hello everyone,
I have a self-hosted GitLab server in my home office that has been working well for several years.
Back in October I changed my internet service from Spectrum cable internet to AT&T fiber. Ever since, I get frequent timeout errors when I try to fetch, push, or pull. By frequent, I mean that I often have to retry 5 or 10 times, every time I want to fetch, push, or pull, until it finally works.
The error is this:
“fatal: unable to access ‘https://redacted.git/’: Failed to connect to redacted.com port 443: Timed out”
One strange factor is that this only seems to occur when I’m working from inside the house, while connected to the same network as the Linux box hosting the GitLab server. My collaborators who connect to the server from outside haven’t had problems, and I even tried connecting my laptop to a hotspot from my phone’s LTE network connection and the problem went away…
I’m not 100% certain if the problem has to do with changing internet service, or if it was due to GitLab version updates performed around the same time.
Anyone have any ideas? The problem is very annoying and wastes a lot of my time, so I would love to resolve it.
Does the name you are using to connect with resolve internally or externally? If externally, the issue you could be having is because the traffic goes out of your internet connection and then comes back in again. Ideally, it would be better to have the hostname resolve to the internal IP of your server to save this from happening. At least that way we can rule out your internet connection.
You can do this temporarily by editing the hosts file on your machine where you are accessing gitlab from (don’t change on your Linux Gitlab server). So if your machine is Windows, edit the c:\windows\system32\drivers\etc\hosts file and put:
x.x.x.x mygitlab.domain.com
obviously changing mygitlab.domain.com for the redacted domain. The x.x.x.x replace with the internal IP of your Linux gitlab server.
Close your web browser, and then open it, so that it will start to use the hosts file entry and then try connecting via gitlab in the browser, as well as from the command line with git.
If you find it then works, it will mean the issue is the traffic having to go out of your internet connection, and then coming back in again. Possibly firewall rules/nat causing issues, slowing it down maybe even attributed by your ISP change (maybe you need to check/verify the firewall/router config).
If you have the ability to run DNS zones on your router/firewall, then this would be ideal, since then when you are at home, the DNS will resolve to internal IP address, and then when you are out of the house, it will use the normal public DNS IP.
Of course, once you have finished testing, remove the entry you put in the hosts file.
Thank you so much Ian! My domain name resolves externally with a noip.com DDNS, so I figured it had something to do with that traffic leaving my router and then coming back in to the same external IP address, but wasn’t sure of a good way to change that.
I added the domain name and internal IP address to the hosts file and that seems to have totally solved the problem. I tested fetches on a bunch of my repositories and they all completed with one second or so, with no timeouts!
It might be nice to “fix” this issue with a setting in my router (I’ll have to read up on DNS zones and see if my AT&T BGW-320 router supports that), but even if the long term fix is the hosts file setting, that would probably be ok since I primarily use this computer for development.
Thank you again so very much, you were a huge help and saved me a bunch of time in the future!
Most if not all home routers dont run a DNS server but worth checking. Might be possible on something running openwrt although I have a fortigate at home that does do it but then its a small business firewall.
Another alternative would be install and configure bind on your linux server since its running 24x7 and use the ip of this as your DNS to resolve internally when at home. Just configure the zone for your domain and then just configure forwarders so that your DNS server can forward queries for all other domains. You could even configure DHCP on your home router to provide the linux server ip for DNS when you obtain an address.
The hosts file fix will be OK providing your computer doesn’t leave the house. If you then want access when away from home, you’ll have to remember to comment the hosts entry with a hash at the beginning of the line, so that it can go back to resolving via DNS. This is why DNS server would help here, to save you this hassle.
EDIT:
From checking, it seems AT&T seem to lock the settings for DNS on the routers they provide, but can’t be sure as I have no way to check this. Some people have said to disable DHCP on the BGW router, and create your own DHCP server internally and use this instead for providing DHCP and DNS settings. So would possibly be bit of extra work than just some DNS configuration.
Just to follow up for anyone else facing this issue, I wanted to let you know about some additional testing that I did.
I tried temporarily disabling all the firewall options that I could find in the AT&T router configuration webpage, as well as the firewall on the Linux server running my GitLab repository. Unfortunately, even when the firewalls disabled, the problem continued to occur (slow fetch/pull/push access with timeouts), so I’m still not sure exactly what the problem is with the BGW-320 configuration.
So, for me, I’m going to just stick with editing the hosts file as the workaround to my problem. The vast majority of the time I will access my GitLab server from home, and when I occasionally need access from outside I’ll just comment out the line in the hosts file.
Thanks again so much for your help Ian, your assistance was really meaningful to me!