We are running GitLab on an ubuntu 18.04 server behind a restrictive firewall, that allows only connections to defined external servers.
To be able to update GitLab using apt package manager, the full qualified domain name (fqdn) of the update server(s) must be registered in the firewall configuration.
Could anybody provide information on which servers and protocolls are to be allowed in the firewall to be able to process the automatic update procedure using apt?
Any advice is welcome.
The problem is still present and any idea for solution is very welcome.
We temporally managed to get updates by allowing certain IP adresses in the firewall that were observed to be connected for updates. But the problem is that the IPs change frequently, so that this is not a good solution.
Any firewall solution on ubuntu uses the same framework in the Linux kernel, and that only supports ips. So your firewall solution resolves hostnames when setting up rules, but unless that is redone quite frequently, using hostnames that point to changing IP’s will cause problems.
In general: It’s a bad idea to uses hostnames in firewall rules (the software that does the resolving might make things less bad, but not all problems can be avoided), the major problem is that you need to have working DNS in order to set firewall rules, depending on specifics, that might mean nothing on the machine gets up if DNS is down - imaging this
being on the DNS server.
If the IPs change as some kind of load balancing scheme, but all IPs work, you can use the same workaround as I’ve used in a similar case: do a standard DNS lookup of the name, select one IP (if multiple are returned, add that to my firewall rules, and put the name into
/etc/hosts to always resolve to that IP. Yes, that breaks the provider’s load balancing, but held against my security, I know what wins for me.
(For GitLab we don’t have a firewall that restrictive.)
Thanks very much for the answer and shared ideas. The problem is that I don’t know which fqdn to include in my /etc/hosts. Here I give you an example of a failed attempt to retrieve package update information for gitlab because the ip 126.96.36.199 is not allowed in our firewall rules:
ubupm@vpc-inptdat:~$ sudo apt update
Err:8 https://packages.gitlab.com/gitlab/gitlab-ce/ubuntu bionic Release
Could not handshake: Error in the pull function. [IP: 188.8.131.52 443]
The fqdn packages.gitlab.com permanently resolves to 184.108.40.206 - so this isn’t the right fqdn:
ubupm@vpc-inptdat:~$ nslookup packages.gitlab.com
Your idea to allow 220.127.116.11 in the firewall and include it in the /etc/hosts with the proper fqdn would perfectly work for me. Do you have an idea which fqdn to include or how to get this fqdn? The ip 18.104.22.168 resolves to server-13-35-254-104.fra6.r.cloudfront.net … which doesn’t help.
Thanks again and best regards,
That message is probably essential to understanding this. The question is why APT is trying to talk to that. My first thought was if it was trying to fetch some CRL to check the certificate against. Searching for the message gave a lot of results that didn’t help me understand what is going on - reading a little further, I found links talking about cloudfront, as you also see involved. But that was an old problem that was solved.
Do you have other lines in
apt/sources.list that this might be related to, and just a confusing ordering of apt’s output?
Do you have a proxy configured for APT?
Thanks again for your helpful answer!
No, I have no other package sources that could be responsible.
Here is an extended output from apt update, to be sure that gitlab makes the problem:
Hit:1 http://ppa.launchpad.net/ondrej/php/ubuntu bionic InRelease
Hit:2 http://ppa.launchpad.net/webupd8team/java/ubuntu bionic InRelease
… all de.archive.ubuntu.com …
Ign:15 https://packages.gitlab.com/gitlab/gitlab-ce/ubuntu bionic InRelease
Get:16 http://security.ubuntu.com/ubuntu bionic-security/main i386 Packages [403 kB]
Get:17 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages [573 kB]
Err:18 https://packages.gitlab.com/gitlab/gitlab-ce/ubuntu bionic Release
Could not handshake: Error in the pull function. [IP: 22.214.171.124 443]
Get:19 http://security. ubuntu.com/ubuntu bionic-security/main amd64 DEP-11 Metadata [38,5 kB]
… further from security.ubuntu.com
Reading package lists… Done
E: The repository ‘https://packages. gitlab.com/gitlab/gitlab-ce/ubuntu bionic Release’ no longer has a Release file.
N: Updating from such a repository can’t be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.
It really seems that it has to do with gitlab. Now, the ip is different from the previous one (and that makes the problem). I have no proxy configured on the system.
Just found this one: https://gitlab.com/gitlab-org/omnibus-gitlab/issues/4431
Here, it is mentioned by DJ Mountney that cloudfront is used by gitlab…
The update works like a charm when allowing the ip 126.96.36.199 in the firewall and including the following line in /etc/hosts:
Thanks @grove for useful hints!