GitLab Update Server fqdn

We are running GitLab on an ubuntu 18.04 server behind a restrictive firewall, that allows only connections to defined external servers.

To be able to update GitLab using apt package manager, the full qualified domain name (fqdn) of the update server(s) must be registered in the firewall configuration.

Could anybody provide information on which servers and protocolls are to be allowed in the firewall to be able to process the automatic update procedure using apt?

Any advice is welcome.

Cheers, Markus

The problem is still present and any idea for solution is very welcome.
We temporally managed to get updates by allowing certain IP adresses in the firewall that were observed to be connected for updates. But the problem is that the IPs change frequently, so that this is not a good solution.

Thank you
Markus

Any firewall solution on ubuntu uses the same framework in the Linux kernel, and that only supports ips. So your firewall solution resolves hostnames when setting up rules, but unless that is redone quite frequently, using hostnames that point to changing IP’s will cause problems.

In general: It’s a bad idea to uses hostnames in firewall rules (the software that does the resolving might make things less bad, but not all problems can be avoided), the major problem is that you need to have working DNS in order to set firewall rules, depending on specifics, that might mean nothing on the machine gets up if DNS is down - imaging this
being on the DNS server.

If the IPs change as some kind of load balancing scheme, but all IPs work, you can use the same workaround as I’ve used in a similar case: do a standard DNS lookup of the name, select one IP (if multiple are returned, add that to my firewall rules, and put the name into /etc/hosts to always resolve to that IP. Yes, that breaks the provider’s load balancing, but held against my security, I know what wins for me.

(For GitLab we don’t have a firewall that restrictive.)

Thanks very much for the answer and shared ideas. The problem is that I don’t know which fqdn to include in my /etc/hosts. Here I give you an example of a failed attempt to retrieve package update information for gitlab because the ip 13.35.254.104 is not allowed in our firewall rules:

ubupm@vpc-inptdat:~$ sudo apt update

Err:8 https://packages.gitlab.com/gitlab/gitlab-ce/ubuntu bionic Release
Could not handshake: Error in the pull function. [IP: 13.35.254.104 443]

The fqdn packages.gitlab.com permanently resolves to 54.153.54.194 - so this isn’t the right fqdn:

ubupm@vpc-inptdat:~$ nslookup packages.gitlab.com
Server: 127.0.0.53
Address: 127.0.0.53#53

Non-authoritative answer:
Name: packages.gitlab.com
Address: 54.153.54.194

Your idea to allow 13.35.254.104 in the firewall and include it in the /etc/hosts with the proper fqdn would perfectly work for me. Do you have an idea which fqdn to include or how to get this fqdn? The ip 13.35.254.104 resolves to server-13-35-254-104.fra6.r.cloudfront.net … which doesn’t help.

Thanks again and best regards,
Markus

That message is probably essential to understanding this. The question is why APT is trying to talk to that. My first thought was if it was trying to fetch some CRL to check the certificate against. Searching for the message gave a lot of results that didn’t help me understand what is going on - reading a little further, I found links talking about cloudfront, as you also see involved. But that was an old problem that was solved.

Do you have other lines in apt/sources.list that this might be related to, and just a confusing ordering of apt’s output?

Do you have a proxy configured for APT?

Thanks again for your helpful answer!

No, I have no other package sources that could be responsible.
Here is an extended output from apt update, to be sure that gitlab makes the problem:

Hit:1 http://ppa.launchpad.net/ondrej/php/ubuntu bionic InRelease
Hit:2 http://ppa.launchpad.net/webupd8team/java/ubuntu bionic InRelease
… all de.archive.ubuntu.com
Ign:15 https://packages.gitlab.com/gitlab/gitlab-ce/ubuntu bionic InRelease
Get:16 http://security.ubuntu.com/ubuntu bionic-security/main i386 Packages [403 kB]
Get:17 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages [573 kB]

Err:18 https://packages.gitlab.com/gitlab/gitlab-ce/ubuntu bionic Release
Could not handshake: Error in the pull function. [IP: 13.32.222.165 443]
Get:19 http://security. ubuntu.com/ubuntu bionic-security/main amd64 DEP-11 Metadata [38,5 kB]
… further from security.ubuntu.com
Reading package lists… Done
E: The repository ‘https://packages. gitlab.com/gitlab/gitlab-ce/ubuntu bionic Release’ no longer has a Release file.
N: Updating from such a repository can’t be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.

It really seems that it has to do with gitlab. Now, the ip is different from the previous one (and that makes the problem). I have no proxy configured on the system.

Best,
Markus

Just found this one: https://gitlab.com/gitlab-org/omnibus-gitlab/issues/4431
Here, it is mentioned by DJ Mountney that cloudfront is used by gitlab…

Maybe the solution: https://packages.gitlab.com/gitlab/gitlab-ce/ubuntu/dists/bionic/Release is forwarded to https://d20rj4el6vkp4c.cloudfront.net/7/8/ubuntu/dists/bionic/

So, d20rj4el6vkp4c.cloudfront.net may be the required fqdn?! I will test this one.

The update works like a charm when allowing the ip 13.224.197.30 in the firewall and including the following line in /etc/hosts:

13.224.197.30 d20rj4el6vkp4c.cloudfront.net

Thanks @grove for useful hints!

1 Like