Gitlab Runner: x509: certificate signed by unknown authority

Hi, I am trying to get my docker registry running again.
First my setup: The Gitlab WebGUI is behind a reverse proxy (ports 80 and 443). The SSH Port for cloning and the docker registry (port 5005) are bind to my public IPv4 address. I have a lets encrypt certificate which is configured on my nginx reverse proxy. My gitlab runs in a docker environment.
Now I tried to configure my docker registry in “gitlab.rb” to use the same certificate. To do that I copied the fullchain.pem and privkey.pem to mydomain.crt and mydomain.key under /etc/gitlab/ssl.

This is what I configured in “gitlab.rb”:

registry_external_url 'https://mydomain:5005'
gitlab_rails['registry_enabled'] = true
gitlab_rails['registry_host'] = "mydomain"
gitlab_rails['registry_port'] = "5005"
gitlab_rails['registry_path'] = "/var/opt/gitlab/gitlab-rails/shared/registry"

registry_nginx['enable'] = true
registry_nginx['ssl_certificate'] = "/etc/gitlab/ssl/mydomain.crt"
registry_nginx['ssl_certificate_key'] = "/etc/gitlab/ssl/mydomain.key"

When I try to login with docker or try to let a runner running (I already had gitlab registry in use but then I switched to reverse proxy and also changed the domain) I get the following error:

ERROR: Preparation failed: Error response from daemon: Get https://mydomain:5005/v2/: x509: certificate signed by unknown authority (executor_docker.go:175:0s)

I also have read the documentation on Container Registry in Gitlab (https://docs.gitlab.com/ee/administration/packages/container_registry.html#configure-container-registry-under-its-own-domain) and tried the Troubleshooting steps. The problem here is that the logs are not very detailed and not very helpful.

Does anybody know what I am doing wrong?

Hi,

this sounds as if the registry/proxy would use a self-signed certificate. Verify that by connecting via the openssl CLI command for example.

openssl s_client -showcerts -connect mydomain:5005

If that’s the case, verify that your Nginx proxy really uses the correct certificates for serving 5005 via proxypass.

Cheers,
Michael

First error:

verify error:num=20:unable to get local issuer certificate

But the certificate is the right one:

issuer=C = US, O = Let's Encrypt, CN = Let's Encrypt Authority X3

( I deleted the rest of the output but compared the two certs and they are the same)

Hm, maybe Nginx doesn’t include the full chain required for validation. How do the portions in your Nginx config look like for adding the certificates? You may need the full pem there.

Cheers,
Michael

@dnsmichi
It hasn’t something to do with nginx. The ports 80 and 443 which are redirected over the reverse proxy are working.
The thing that is not working is the docker registry which is not behind the reverse proxy. It is bound directly to the public IPv4.

Ah, I see. Typical Monday where more coffee is needed

Then I would inspect whether only the .crt is enough for the configuration, of if you can use the pull PEM in that path, including the certificate chain.

registry_nginx['ssl_certificate'] = "/etc/gitlab/ssl/mydomain.crt"

Cheers,
Michael

@dnsmichi
I mentioned in my question that I copied fullchain.pem to /etc/gitlab/ssl/mydomain.crt and privkey.pem to mydomain.key. Is that the correct what I’ve done?

I also showed my config for registry_nginx where I give the path to the crt and the key. I am sure that this is right

It should be correct, that was a missing detail. I’m wondering though why the runner doesn’t pick it up, set aside from the openssl connect.

Did you register the runner before with a custom --tls-ca-file parameter before, shown here?

It should be seen in the runner config.toml, can you look for that specific setting (likewise, post the config from the runner without sensitive details).

Cheers,
Michael

@dnsmichi Sorry I forgot to mention that also a docker login is not working. I get the same result there as with the runner.

No worries, the more details we unveil together, the better.

I’d suggest using sslscan and run a full scan on your host.

sslscan mydomain:5005

This should provide more details about the certificates, ciphers, etc.

Cheers,
Michael

So this is what I get:

Version: 1.11.5
OpenSSL 1.0.2n  7 Dec 2017

OpenSSL version does not support SSLv2
SSLv2 ciphers will not be detected

OpenSSL version does not support SSLv3
SSLv3 ciphers will not be detected
Testing SSL server mydomain on port 5005

  TLS renegotiation:
Session renegotiation not supported

  TLS Compression:
OpenSSL version does not support compression
Rebuild with zlib1g-dev package for zlib support

  Heartbleed:
TLS 1.2 not vulnerable to heartbleed
TLS 1.1 not vulnerable to heartbleed
TLS 1.0 not vulnerable to heartbleed

  Supported Server Cipher(s):
Preferred TLSv1.2  256 bits  ECDHE-RSA-AES256-GCM-SHA384   Curve P-256 DHE 256
Accepted  TLSv1.2  128 bits  ECDHE-RSA-AES128-GCM-SHA256   Curve P-256 DHE 256
Accepted  TLSv1.2  256 bits  ECDHE-RSA-AES256-SHA384       Curve P-256 DHE 256
Accepted  TLSv1.2  128 bits  ECDHE-RSA-AES128-SHA256       Curve P-256 DHE 256
Accepted  TLSv1.2  256 bits  ECDHE-RSA-AES256-SHA          Curve P-256 DHE 256
Accepted  TLSv1.2  128 bits  ECDHE-RSA-AES128-SHA          Curve P-256 DHE 256
Accepted  TLSv1.2  256 bits  AES256-GCM-SHA384
Accepted  TLSv1.2  128 bits  AES128-GCM-SHA256
Accepted  TLSv1.2  256 bits  AES256-SHA256
Accepted  TLSv1.2  128 bits  AES128-SHA256
Accepted  TLSv1.2  256 bits  AES256-SHA
Accepted  TLSv1.2  128 bits  AES128-SHA

  SSL Certificate:
Signature Algorithm: sha256WithRSAEncryption
RSA Key Strength:    2048

Subject:  mydomain
Altnames: DNS:mydomain
Issuer:   Let's Encrypt Authority X3

Not valid before: Nov 24 20:57:50 2019 GMT
Not valid after:  Feb 22 20:57:50 2020 GMT

Ok, we are getting somewhere. You may see a German Telekom IP address in your logs, I’d suggest editing the web host above in your output.

So it is indeed the full chain missing in the certificate. I remember having that issue with Nginx a while ago myself.

You’re saying that you have the fullchain.pem and privkey.pem from Let’s Encrypt. Keep their names in the config, I’m not sure if that file suffix makes a difference.

mkdir -p /etc/gitlab/ssl/mydomain
cp fullchain.pem /etc/gitlab/ssl/mydomain/
cp privkey.pem /etc/gitlab/ssl/mydomain/

Then modify the config like this:

registry_nginx['ssl_certificate'] = "/etc/gitlab/ssl/mydomain/fullchain.pem"
registry_nginx['ssl_certificate_key'] = "/etc/gitlab/ssl/mydomain/privkey.pem"

Then reconfigure/restart GitLab.

@dnsmichi Thanks I forgot to clear this one

@dnsmichi hmmm we seem to have got an step further:
This is the error message when I try to login now:

Error response from daemon: Get https://mydomain:5005/v2/: denied: access forbidden

Next guess: File permissions. Ensure that the GitLab user (likely git) owns these files, and that the privkey.pem is also chmod 400.

For the login you’re trying, is that something like this?

CI_JOB_TOKEN=xxx
docker login -u gitlab-ci-token -p $CI_JOB_TOKEN mydomain:5005

@dnsmichi To answer the last question: Nearly yes. I am trying docker login mydomain:5005 and then I get asked for username and password.
I will show after the file permissions.

@dnsmichi My gitlab is running in a docker container so its the user root to whom it should belong. Am I right?

Found a little message in /var/log/gitlab/registry/current:

Error authorizing context: authorization token required

I don’t have enabled 2FA so I am a little bit confused. Or does this message mean another thing?

Within the CI job, the token is automatically assigned via environment variables. For your tests, you’ll need your username and the authorization token for the API. You can create that in your profile settings.

05526×778 28.2 KB

Then use

docker login -u yourusername -p youraccesstoken mydomain:5005

@dnsmichi is this new? A few versions before I didn’t needed that.
But this is not the problem. I generated a code with access to everything (after only api didn’t work) and it is still not working.

(I posted to much for my first day here so I had to wait :D)