Gitlab not able to verify AD certificate for authentication

I have a weird issue when connecting my gitlab instance to Active Directory. Unfortunately the AD is not managed by me and there’s nothing I can change at this point, so I’ll have to work with what I have, but the AD doesn’t offer a higher TLS version than 1.0 (I know ). I’ve tested it with testssl and I’ve checked the pcap (wireshark) and that confirms it.

The point is, the certificate works without any issues when I try to query the AD directly with ldapsearch (and adding it to the ldap.conf) and I’m forcing starttls, but for some reason, gitlab cannot verify the authenticity of the certificate and it throws this error.

Could not authenticate you from Ldapmain because “Ssl connect returned=1 errno=0 state=error: certificate verify failed (unable to get local issuer certificate)”.

So I’m sure the certificate can be recognized as valid despite the tls crappy standard

I’ve already added the certificate to gitlab and it has created a symlink pointing to it in /etc/gitlab/trusted-certs, so this part is covered.

Any ideas what I might be missing?

I’m using gitlab 14.1.1

I think you can put

verify_certificates = false

Or something along those lines in gitlab.rb. Otherwise you need the cacert exporting from AD and configured on your gitlab server for capath to get it to work if you dont want to ignore certificate verification.

Maybe I wasn’t explicit enough, but this is exactly the reason why I’m posting here - to understand why gitlab doesn’t recognize the certificate, not the find ways of avoiding the issue (verifiy false).

I don’t think I need the CA. Using the certificate which AD presents should be enough. Again, ldapsearch recognizes it as such. It normally complains if it thinks the certificate is not valid.

So if anyone has any knowledge in this direction, I’d be greatful.

There are other posts on this forum about it. If you do not have a purchased certificate verified by an external authority, then it will complain about your Active Directory certificate since the default one is not verified by an authority. It is generated internally, so effectively self-signed.

Other people have posted about this. You have two ways around it, either use the option to not verify the certificate, or get the CA certificate from your Active Directory domain, and configure this in gitlab.rb so that the verify errors don’t appear - since the exported CA certificate will then verify the certificate what is connected to the AD domain. It might even be enough just to put this CA certificate in /get/gitlab/trusted-certs and nothing else will need to be done.

So you are wrong, you do need the CA since it’s an internally generated certificate and a publicly available CA isn’t able to verify your certificate. Alternatively, disable the verify function so that you don’t need to use the CA. Not all programs complain about internally generated certificates. And some do, like gitlab in this instance.

The good thing you have with gitlab though, is you have options to ignore the errors, or provide the CA certificate from the domain.

Other products, eg: from RSA which I have used you must give the internal CA cert, since it won’t let you override with an option like gitlab.

Of course this is a self-signed certificate.

When I enter a website which uses a self-sign certificate and I accept the certificate (I use the exception), this certificate is imported into the browser (in firefox, for example, or for Safari in the certificate store/key chain, whatever - mac os even asks for the user’s password in this case).
The browser/operating system has no idea what the CA of that certificate is. It doesn’t need to. It accepts only the certificate itself. Next time I visit the website, the browser doesn’t complain anymore.
But what you’re saying basically is that out of principle (as a self-understood statement!) you cannot accept the certificate, but only the CA and this should be taken for granted.

Again, ldapsearch uses /etc/ldap.conf and has a directive TLS_CACERT:

TLS_CACERT	/usr/local/share/ca-certificates/my_ad_certificate.pem

I use -ZZ to make sure it fails if the tls connection doesn’t work properly. When it doesn’t point to the AD certififcate, I immediately get:

ldap_start_tls: Connect error (-11)
	additional info: (unknown error code)

So no, I don’t think I’m wrong, meaning that importing the certificate itself instead of the CA which generated it should normally be enough. If gitlab behaves differently, then that should be explicitly stated, but I think the argument behind it is wrong, because it feels like it’s based on the wrong a assumption.
I know the cleanest way is to simply use the CA and I’m fine with that, but under the circumstances I can’t obtain it. It is what it is.

[edit:] Ignoring the authenticity of the certificate even in these poor circumstances kind of beats the purpose of security to a substantial extent, despite the fact that the encryption itself does work.
The less control you have over your environment (over the media between the AD and gitlab), the worse it is.

Whether you import the certificate and accept the risk (Firefox), or whether you say verify_certificate = false in gitlab, it’s the same thing since you are ignoring the fact that you aren’t comparing the certificate with the CA that generated it.

As I said to you, there are plenty of other solutions, from RSA Security for example, as well as others that don’t let you ignore the certificate verify problem so you either have to purchase a commercial certificate, or provide the CA certificate for the internally generated one.

The same happens if I run my own internal CA. All certificates I generate must be accompanied with the exported CA certificate from my internal CA for them to be trusted. I can import the CA certificate into Firefox, and then every single certificate I generate in the future will be trusted by Firefox. Otherwise, I would have to import hundreds (accepting the untrusted certificates) of internally generated certificates for my system. By doing the CA, I just import one.

Not all applications let you import the certificate itself, they are configured to use the CA certificate to verify against the system that it is attempting to connect to. And this is by far better a method than the one you are suggesting which is a far worse method.

You contradict yourself in your last paragraph by stating:

ignoring the authenticity of the certificate even in these poor circumstances kind of beats the purpose of security

so importing the certificate is safer than not verifying it? It’s exactly the same thing since you have no way to trust it. Verifying against the CA is more secure.

You are right in principle about all this. But the point is, if I know I get the certificate from a correct source and I can make sure it’s authentic, then technically that’s enough.
It obviously makes more sense to distribute the certificates correctly through an internal CA etc., etc. That’s not the point.

I want to understand technically why gitlab doesn’t accept something which basically all other applications accept - importing the certificate itself instead of the CA. I’m not even ruling out that there might be something wrong with the AD/certificate itself, but I don’t understand why it works in other circumstances.

Whether you import the certificate and accept the risk (Firefox), or whether you say verify_certificate = false in gitlab it’s the same thing since you are ignoring the fact that you aren’t comparing the certificate with the CA that generated it.

Well yeah, but if you have a way of knowing that that certificate is actually authentic (you receive it on a safe medium from someone you know), then that’s enough. You accept that certificate as is, end of story, impractical as it may be. But technically there’s no difference as to the validity of the certificate.

Probably similar like the RSA products I’ve used - for security reasons, best practices for certificate integration since the proper method for verifying certificates is with the CA. Also, as I mentioned RSA don’t even give you the ability to ignore verification. If you don’t verify the internal certificate with a CA, then you cannot use their product at all. So Gitlab is more forgiving in this instance. That’s just the way it is.

Personally I prefer products that do things properly than some hack with ignoring certificates or falsely accepting them like with what happens with Firefox when connecting to untrusted sites. It’s far more professional.

I think it’s about of striking the right balance and not simply enforcing things on users, or better yet, on administrators.
If you want to test a web service on your local computer, you shouldn’t really need to import a CA and generate certificates for each one, you just want to use that service locally and you do that understanding the circumstances. In an enterprise/company environment it’s a different story, of course.

I don’t understand the difference in this respect between accepting a certificate as an exception, as firefox does, and not verifying the authenticity of the certificate in the way gitlab allows it to. I don’t find that “unprofessional”, I think it’s very important that they both exist as options
The option of ignoring the authenticity of a certificate is widespread I would say (I can come up with dozens of examples), but you normally have to enforce it - which I think is correct.

@lethargos this is a great conversation between you and @iwalker. @lethargos, certificates have become much more complicated in the last five years. @iwalker, there are hopefully other configuration options such as broadening the scope of accepted certificates as opposed to dropping them altogether. Where does one set certificate constraint options for gitlab?

Teasing out exactly where certificate problems lie is hard. We have a Linux VM that is our root level CA which only generates intermediate CAs. The idea is that the root is completely offline. One intermediate CA for all other *nix machines and another intermediate CA for MS ActiveDirectoryServices. The public root certificate and the intermediate certificate are both required by most applications to work properly. In other words, the entire “Certificate Chain” is required. Various vendors have dramatically changed requirements for SSL/TLS and the constraints on the certificate chain. For instance, Apple says it will no longer accept certificates that last more than about a year. Google Chrome and Firefox diverged significantly in where and how they handle certificates.

In your Firefox example, the self-signed certificate did not chain up to anywhere else or the certificate included the entire chain. In other words, no CA was involved or the CA was included in the certificate.

The TLS_CACERT directive certainly looks like a Certificate Authority directive, not just a single downlevel certificate. The CA certificate you need for AD must already be installed in your AD joined systems. If your existing my_ad_certificate.pem does not include the chain up to your primary domain controller, then on your MS Windows PC, open certlm.msc and look for the root CA there. Put that in TLS_CACERT.

There could be many other reasons the certificate is failing. It might not have enough bits or not digitally signed with an old and no longer trusted algorithm. For instance, some web browsers will not trust any homemade certificate chain if the root is signed with SHA1. A SHA256 root will work fine. For instance, we have old hardware that java does not trust the certificate because it is only 1024 bits. Telling java to accept certificates of 1024 or greater and it works fine. Google Chrome may accept a longstanding well known root CA signed with SHA1, but will not accept a locally generated infrastructure of the same type. SHA256 is required from the top down.

The certificate may not have roles needed for the what is expected of the operation. For instance, a valid personal certificate does not work for code signing unless that role is stipulated in the certificate chain.

Hi,

Can you install sslscan (apt install sslscan or brew install sslscan) and run it against your LDAP server to see the certificates and TLS versions being offered? Alternatively, you can use openssl s_client -connect address:port.

Also, can you share the LDAP settings from the gitlab.rb configuration file (obfuscated)?

Cheers,
Michael

Hi, dnsmichi. Thanks for the reply. Here are the configuration for LDAP and the result of openssl:

gitlab_rails['ldap_servers'] = YAML.load <<-'EOS'
main: # 'main' is the GitLab 'provider ID' of this LDAP server
     label: 'Active Directory'
     host: 'dc1.mydomain.company.com'
     port: 636
     uid: 'sAMAccountName'
     encryption: 'simple_tls'
     bind_dn: 'CN=gitlab,OU=GitLabCom,OU=Special Users,DC=mydomain,DC=company,DC=com'
     password: 'mypassword'
     active_directory: true
     verify_certificates: true
     tls_options:
       ca_file: /usr/local/share/ca-certificates/dc1_cert.pem
       ssl_version: 'TLSv1'
     allow_username_or_email_login: false
     block_auto_created_users: false
     base: 'OU=Users,OU=Internal,DC=mydomain,DC=company,DC=com'

setting verifiy_certificates to false makes it work immediately and and TLS 1.0 is negociated.
openssl says:

CONNECTED(00000003)
depth=0
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0
verify error:num=21:unable to verify the first certificate
verify return:1
---
Certificate chain
 0 s:
   i:/DC=COM/DC=COMPANY/DC=MYDOMAIN/CN=CompanyCA
---
Server certificate
-----BEGIN CERTIFICATE-----
*thecertificate - removed*
-----END CERTIFICATE-----
subject=
issuer=/DC=COM/DC=COMPANY/DC=MYDOMAIN/CN=CompanyCA
---
Acceptable client certificate CA names
/CN=DC1.MYDOMAIN.COMPANY.COM
/DC=COM/DC=Company/DC=MYDOMAIN/CN=CompanyCA

/DC=COM/DC=COMPANY/DC=MYDOMAIN/CN=companyca
/C=SE/O=AddTrust AB/OU=AddTrust External TTP Network/CN=AddTrust External CA Root
/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert Assured ID Root CA
/C=US/O=The Go Daddy Group, Inc./OU=Go Daddy Class 2 Certification Authority
/C=US/O=GeoTrust Inc./CN=GeoTrust Primary Certification Authority
/C=US/O=VeriSign, Inc./OU=VeriSign Trust Network/OU=(c) 2008 VeriSign, Inc. - For authorized use only/CN=VeriSign Universal Root Certification Authority
/C=US/O=VeriSign, Inc./OU=VeriSign Trust Network/OU=(c) 2006 VeriSign, Inc. - For authorized use only/CN=VeriSign Class 3 Public Primary Certification Authority - G5
/O=Entrust.net/OU=www.entrust.net/CPS_2048 incorp. by ref. (limits liab.)/OU=(c) 1999 Entrust.net Limited/CN=Entrust.net Certification Authority (2048)
/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert High Assurance EV Root CA
/C=US/O=VeriSign, Inc./OU=Class 3 Public Primary Certification Authority
/OU=GlobalSign Root CA - R2/O=GlobalSign/CN=GlobalSign
/C=US/O=Entrust, Inc./OU=See www.entrust.net/legal-terms/OU=(c) 2009 Entrust, Inc. - for authorized use only/CN=Entrust Root Certification Authority - G2
/C=US/O=GTE Corporation/OU=GTE CyberTrust Solutions, Inc./CN=GTE CyberTrust Global Root
/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert Global Root CA
/C=US/O=Starfield Technologies, Inc./OU=Starfield Class 2 Certification Authority
/C=BE/O=GlobalSign nv-sa/OU=Root CA/CN=GlobalSign Root CA
/C=US/O=Entrust, Inc./OU=www.entrust.net/CPS is incorporated by reference/OU=(c) 2006 Entrust, Inc./CN=Entrust Root Certification Authority
/C=GB/ST=Greater Manchester/L=Salford/O=Comodo CA Limited/CN=AAA Certificate Services
/C=IE/O=Baltimore/OU=CyberTrust/CN=Baltimore CyberTrust Root
/O=Digital Signature Trust Co./CN=DST Root CA X3
/C=US/O=GeoTrust Inc./CN=GeoTrust Global CA
/C=US/ST=Washington/L=Redmond/O=Microsoft Corporation/CN=Microsoft Root Certificate Authority 2010
/C=US/ST=Washington/L=Redmond/O=Microsoft Corporation/CN=Microsoft Root Certificate Authority 2011
/OU=Copyright (c) 1997 Microsoft Corp./OU=Microsoft Corporation/CN=Microsoft Root Authority
/DC=com/DC=microsoft/CN=Microsoft Root Certificate Authority
/CN=NT AUTHORITY
Client Certificate Types: RSA sign, DSA sign, ECDSA sign
---
SSL handshake has read 4841 bytes and written 627 bytes
---
New, TLSv1/SSLv3, Cipher is AES128-SHA
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1
    Cipher    : AES128-SHA
    Session-ID: D82E0000C4E68E856451ECAC0519F794B643E09B56A44CB91B3548BB1EECCD2F
    Session-ID-ctx:
    Master-Key: 91736D82DB6B10CF5F752ED3A7BE92A923447B2D278F852918343AFF4BC8F4C1888723FA1BD85859B5C68F46E6A59D70
    Key-Arg   : None
    Krb5 Principal: None
    PSK identity: None
    PSK identity hint: None
    Start Time: 1628189269
    Timeout   : 300 (sec)
    Verify return code: 21 (unable to verify the first certificate)

In the certificate headers I have:

            X509v3 Subject Alternative Name: critical
                DNS:DC1.MYDOMAIN.COMPANY.COM, DNS:MYDOMAIN.COMPANY.COM, DNS:MYDOMAIN

@FossCoder Thanks for the insight. This is very interesting. If you have any other concrete examples from your experience of how certain browers react to certain scenarios, I’d be happy to hear. Even for example how FF and Chrome diverged from how Apple handles certificates. I thought the 13-month period has become generally accepted, and I’m not sure you can buy a commercial certificate with longer for a period longer than 13 months. And if you do, I know you might get in trouble (theoretically). This has come into effect September 2020, if I remember correctly, that is to say, only for certificates which were issued from September onwards.

Certificate Authorities and Web Browser Interoperability Forum Baseline Requirements Documents (SSL/TLS Server Certificates) | CAB Forum
For some reason, i cannot find the emails from way back, but the CABforum is how web browser developers such as Mozilla and certificate authorities, ie: comodo or digicert, discuss and agree on constraints. These baseline requirements are updated frequently.

Hi,

Unfortunately this isn’t a good idea for production, from a security standpoint.

Out of curiosity, did you run update-ca-certificates after copying the ca_file certificate there? Might have an impact how this is treated by the OS making it globally available.

Also, this path seems to coming from Debian/Ubuntu - can you share your OS details?

$ cat /etc/os-release

Another thing you can test is with removing the ssl_version key in the LDAP YAML config, and see what happens. Maybe the pinning does not work reliably. Typically server and client negotiate on the best version.

Another guess of mine - symlinks might not be followed when reading certificate paths. Therefore I’d recommend the following steps for a self-signed certificate and CA:

$ cp /usr/local/share/ca-certificates/dc1_cert.pem /etc/gitlab/trusted-certs/

$ gitlab-ctl reconfigure

Following this topic, Perl might be a requirement for above steps. Ubuntu gitlab-ee-omnibus ssl ldap certificate verify failed - #3 by itkroplis

Cheers,
Michael

Hi, dnsmichi. Thanks for the answer.

Unfortunately this isn’t a good idea for production, from a security standpoint.

I mentioned that myself earlier in the thread

I only added ssl_version afterwards, to force gitlab to use it, in case it was trying to use another protocol while verifiy_certificate is true. Didn’t help a lot.

Out of curiosity, did you run update-ca-certificates after copying the ca_file certificate there?

ca_file was also added afterwards, just to try to explicitly point it to the certifcate. Yes, I’ve already run update-ca-certificates to no effect. What I did from the very beginning was to add the certificate in /etc/gitlab/trusted-certs. The symlink is being created (I mention this in my initial post), so that part seems to work correctly in gitlab.

I’m using Ubuntu 20.04.2

I also followed the thread you pointed to, and also the other links. Doesn’t seem to apply to me, because the symlink is being created and I already have perl installed anyway.

gitlab-ctl reconfigure will not do the job if certificate files were changed, but not configuration files.

sudo gitlab-ctl hup nginx to cause NGINX to reload the existing configuration and new certificates gracefully.

https://docs.gitlab.com/omnibus/settings/nginx.html#update-the-ssl-certificates

I Have to say, I’m not sure what this has to do with my problem
There’s nothing wrong with nginx. It presents the correct certificate, it’s a different service, a different context.

Both of your errors are mentioned in the “Common SSL errors” section

As mentioned before, ssl often needs need your local AD certificate to have the entire chain, meaning it not only includes your gitlab host/user certificate, but also any intermediate certs, the CA cert, and the root certificate.

Lastly, since gitlab has its own version of ssl, the docs mention how to test it.

echo | /opt/gitlab/embedded/bin/openssl s_client -connect dcIPaddress:636 | /opt/gitlab/embedded/bin/openssl x509 -text -noout

Yes, thanks for that, I didn’t know you could test it directly with their version of openssl. The result is the same as the openssl of the OS.

But yeah, I’m not sure if I can do anything else without getting the CA/intermediate CA from the AD and/or trying to suggest some changes on the AD’s side (not going to happen:) ).