Self Hosted Installation Error Help

I’m trying to install GitLab on a new Ubuntu 20.04.4 server following the instructions here:

When I get to Step 2 and issue the following command:

curl https://packages.gitlab.com/install/repositories/gitlab/gitlab-ee/script.deb.sh | sudo bash

I get this error:

curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: curl - SSL CA Certificates

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

It looks like I can do a “dpkg-reconfigure ca-certificates” command, but should that really be necessary?

Does your CA certs on the ubuntu server have the CA thats signed the the gitlab cert?

06:55:01 ubuntu@vps-f116ed9f certs → ls -lrt bal*
lrwxrwxrwx 1 root root 64 May 18 2021 Baltimore_CyberTrust_Root.pem → /usr/share/ca-certificates/mozilla/Baltimore_CyberTrust_Root.crt

1 Like

apt can be configured to not run the configure steps automatically after installing a package. That may need a manual step.

IIRC - not 100% sure - certain cloud provider images for Ubuntu/Debian do not update the CA certificate cache, and you need to do that manually after booting the VM (or have that automated in your IaC tools, like an Ansible playbook).

TL;DR - running dpkg-reconfigure ca-certificates won’t do harm to the system, alternatively update-ca-certificates.

Yes, that file/certificate is there.

I did try running “sudo update-ca-certificates”, but it didn’t do anything (0 added, 0 removed).

When I run “sudo dpkg-reconfigure ca-certificates” and choose “ask” there are quite a few and they all look selected ([*]). I didn’t try just selecting “yes”, , and running update-ca-certificates because I wasn’t sure if that is typically necessary on a new server.

Hmmm, tricky. Can you share which cloud provider and/or base image you are using, and the steps to get a running VM with Ubuntu 20? I’d like to see if we can reproduce the error, or otherwise give better suggestions from our experience.

It does seem like that curl does not pick up the root CA certificates stored in the OS. Can you run the following command and post its output please:

curl --version

Another guess - I have seen the error message

curl: (60) SSL certificate problem: unable to get local issuer certificate

also with incompatible TLS ciphers and versions. curl can be run in debug verbose mode with multiple -v parameters, please try that and post the output here too.

curl -vvvv https://packages.gitlab.com/install/repositories/gitlab/gitlab-ee/script.deb.sh

The server is running on VMWare EXSI managed with VSphere. Ubuntu was installed from the stock 20.04.4 LTS server image downloaded from the Ubuntu Web site.

curl --version
curl 7.68.0 (x86_64-pc-linux-gnu)

curl -vvvv https://packages.gitlab.com/install/repositories/gitlab/gitlab-ee/script.deb.sh

`

  • Trying 104.18.26.123:443…
  • TCP_NODELAY set
  • Connected to packages.gitlab.com (104.18.26.123) port 443 (#0)
  • ALPN, offering h2
  • ALPN, offering http/1.1
  • successfully set certificate verify locations:
  • CAfile: /etc/ssl/certs/ca-certificates.crt
    CApath: /etc/ssl/certs
  • TLSv1.3 (OUT), TLS handshake, Client hello (1):
  • TLSv1.3 (IN), TLS handshake, Server hello (2):
  • TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
  • TLSv1.3 (IN), TLS handshake, Certificate (11):
  • TLSv1.3 (OUT), TLS alert, unknown CA (560):
  • SSL certificate problem: unable to get local issuer certificate
  • Closing connection 0
    curl: (60) SSL certificate problem: unable to get local issuer certificate
    More details here: curl - SSL CA Certificates

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.
`

What certificate authority is used so I can ask IT if our firewall is possibly blocking something?

Thanks for confirming. So in theory, a minimal official ubuntu Docker image can also be used to follow the steps and make suggestions on things to check.

docker run -ti ubuntu:latest bash
apt update && apt install -y curl

The full version output is interesting to learn which libraries curl is linked against - OpenSSL, or some alternative implementation like libreSSL. Could be a direction to search for bugs.

root@8634df6834f6:/# curl --version
curl 7.68.0 (x86_64-pc-linux-gnu) libcurl/7.68.0 OpenSSL/1.1.1f zlib/1.2.11 brotli/1.0.7 libidn2/2.2.0 libpsl/0.21.0 (+libidn2/2.2.0) libssh/0.9.3/openssl/zlib nghttp2/1.40.0 librtmp/2.3
Release-Date: 2020-01-08
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtmp rtsp scp sftp smb smbs smtp smtps telnet tftp
Features: AsynchDNS brotli GSS-API HTTP2 HTTPS-proxy IDN IPv6 Kerberos Largefile libz NTLM NTLM_WB PSL SPNEGO SSL TLS-SRP UnixSockets

Knowing that OpenSSL comes to play, we can switch the debugging client to openssl and connect to the package server.

Please run the command below also on your server, and post the output. I am sharing mine for comparison. Depending on the server, you may need to run apt -y install openssl first.

root@8634df6834f6:/# openssl s_client -connect packages.gitlab.com:443

CONNECTED(00000003)
depth=2 C = IE, O = Baltimore, OU = CyberTrust, CN = Baltimore CyberTrust Root
verify return:1
depth=1 C = US, O = "Cloudflare, Inc.", CN = Cloudflare Inc ECC CA-3
verify return:1
depth=0 C = US, ST = California, L = San Francisco, O = "Cloudflare, Inc.", CN = gitlab.com
verify return:1
---
Certificate chain
 0 s:C = US, ST = California, L = San Francisco, O = "Cloudflare, Inc.", CN = gitlab.com
   i:C = US, O = "Cloudflare, Inc.", CN = Cloudflare Inc ECC CA-3
 1 s:C = US, O = "Cloudflare, Inc.", CN = Cloudflare Inc ECC CA-3
   i:C = IE, O = Baltimore, OU = CyberTrust, CN = Baltimore CyberTrust Root
---

---

---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: ECDSA
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 2705 bytes and written 391 bytes
Verification: OK
---
New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
Server public key is 256 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---
---
Post-Handshake New Session Ticket arrived:
SSL-Session:
    Protocol  : TLSv1.3
    Cipher    : TLS_AES_256_GCM_SHA384

Next, let’s print the CA Certificate file.

root@8634df6834f6:/# ls -la /etc/ssl/certs/ca-certificates.crt
-rw-r--r-- 1 root root 195453 Jun 10 17:36 /etc/ssl/certs/ca-certificates.crt

root@8634df6834f6:/# awk -v cmd='openssl x509 -noout -subject' '
    /BEGIN/{close(cmd)};{print | cmd}' < /etc/ssl/certs/ca-certificates.crt

The output can be compared to what is offered when connecting to the server, -showcerts for the openssl command.

openssl s_client -showcerts -connect packages.gitlab.com:443

At this point, I strongly believe that there is a HTTP/TLS proxy in front of you, which intercepts the traffic, and thus presents the wrong CA certificate. We can verify that together by analysing your output from the commands above.

Thanks dnsmichi, I haven’t had time to try that yet, been busy fighting other fires.

Can you tell be the CA authority(s) that the GitLab installation needs and I can check with our IT team to see if something it getting blocked or intercepted.