Enabling SSL failing at Validation failed, unable to request certificate

msalmanmasood · March 31, 2021, 9:24am

Running latest gitlab server version 13.10.0 listening on http or 80 port.

Trying to enable the SSL by introducing following configurations in the gitlab.rb:

external_url ‘https://git.mydomain.ltd’
letsencrypt[‘enable’] = true
letsencrypt[‘contact_emails’] = [‘msk@mydomain.ltd’]
nginx[‘redirect_http_to_https’] = true
nginx[‘redirect_http_to_https_port’] = 80

However, reconfiguring attempt failing by throwing following exception:

Running handlers:
There was an error running gitlab-ctl reconfigure:

letsencrypt_certificate[git.mydomain.ltd] (letsencrypt::http_authorization line 5) had an error: RuntimeError: acme_certificate[staging] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/letsencrypt/resources/certificate.rb line 25) had an error: RuntimeError: ruby_block[create certificate for git.mydomain.ltd] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/acme/resources/certificate.rb line 108) had an error: RuntimeError: [git.sahaba.net] Validation failed, unable to request certificate

whereas hostname to IP resolves successfully through internal DNS as well as hosts file also updated with the same host info.

moreover, the /etc/gitlab/ssl/ does have all necessary certs created successfully - still failing.

any clue?

msalmanmasood · March 31, 2021, 10:04am

Compiled Resource:
------------------
# Declared in /opt/gitlab/embedded/cookbooks/cache/cookbooks/letsencrypt/recipes/http_authorization.rb:5:in `from_file'

letsencrypt_certificate("git.mydomain.tld") do
  action [:create]
  updated true
  updated_by_last_action true
  default_guard_interpreter :default
  declared_type :letsencrypt_certificate
  cookbook_name "letsencrypt"
  recipe_name "http_authorization"
  crt "/etc/gitlab/ssl/git.mydomain.tld.crt"
  key "/etc/gitlab/ssl/git.mydomain.tld.key"
  alt_names []
  cn "git.mydomain.tld"
  only_if { #code block }
end

System Info:
------------
chef_version=15.14.0
platform=centos
platform_version=7.9.2009
ruby=ruby 2.7.2p137 (2020-10-01 revision 5445e04352) [x86_64-linux]
program_name=/opt/gitlab/embedded/bin/chef-client
executable=/opt/gitlab/embedded/bin/chef-client

Running handlers:
There was an error running gitlab-ctl reconfigure:

letsencrypt_certificate[git.mydomain.tld] (letsencrypt::http_authorization line 5) had an error: RuntimeError: acme_certificate[staging] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/letsencrypt/resources/certificate.rb line 25) had an error: RuntimeError: ruby_block[create certificate for git.mydomain.tld] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/acme/resources/certificate.rb line 108) had an error: RuntimeError: [git.mydomain.tld] Validation failed, unable to request certificate

msalmanmasood · March 31, 2021, 11:34am

Strange is the http redirect is working and so do the SSL applied… then why the deployment failed in the end - any validation issue that causing this failure?

Thanks.

msalmanmasood · March 31, 2021, 11:36am

For more details around the running gitlab server configurations, refer to this link Post upgrade GitLab-CE 9.1.4 to 13.10.0 opens activity or admin page only

balonik · March 31, 2021, 12:17pm

in order to use the built-in lets encrypt the domain must be reachable from public internet on http and https standard ports

msalmanmasood · April 1, 2021, 7:02am

tried to use the domain reachable over the internet didn’t work as well.

Looks like some validation is failing in the end while rest of the necessary configurations are in place which has enabled the https successfully.

What else can be checked in this case?

balonik · April 1, 2021, 7:27am

You could try to disable the HTTP to HTTPS redirect during initial certificate setup, because Lets Encrypt validation process is over HTTP http://<YOUR_DOMAIN>/.well-known/acme-challenge/<TOKEN> and I am not sure, but the redirect to HTTPS could interfere with the validation process.
Once the validation passed and you have a working setup you can enable the redirect again.

balonik · April 1, 2021, 7:32am

After looking at this issue the redirect actually might be required.

msalmanmasood · April 1, 2021, 7:37am

yes @balonik it didn’t work as well - rather failed with same error message thrown:

Compiled Resource:
------------------
# Declared in /opt/gitlab/embedded/cookbooks/cache/cookbooks/letsencrypt/recipes/http_authorization.rb:5:in `from_file'

letsencrypt_certificate("git.mydomain.ltd") do
  action [:create]
  updated true
  updated_by_last_action true
  default_guard_interpreter :default
  declared_type :letsencrypt_certificate
  cookbook_name "letsencrypt"
  recipe_name "http_authorization"
  crt "/etc/gitlab/ssl/git.mydomain.ltd.crt"
  key "/etc/gitlab/ssl/git.mydomain.ltd.key"
  alt_names []
  cn "git.mydomain.ltd"
  only_if { #code block }
end

System Info:
------------
chef_version=15.14.0
platform=centos
platform_version=7.9.2009
ruby=ruby 2.7.2p137 (2020-10-01 revision 5445e04352) [x86_64-linux]
program_name=/opt/gitlab/embedded/bin/chef-client
executable=/opt/gitlab/embedded/bin/chef-client

Running handlers:
There was an error running gitlab-ctl reconfigure:

letsencrypt_certificate[git.mydomain.ltd] (letsencrypt::http_authorization line 5) had an error: RuntimeError: acme_certificate[staging] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/letsencrypt/resources/certificate.rb line 25) had an error: RuntimeError: ruby_block[create certificate for git.mydomain.ltd] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/acme/resources/certificate.rb line 108) had an error: RuntimeError: [git.mydomain.ltd] Validation failed, unable to request certificate

balonik · April 1, 2021, 7:44am

It looks like something blocks the traffic from Let’s Encrypt servers to your GitLab instance. Your GitLab instance must be reachable over HTTP (TCP/80) from public internet for the validation to pass.

msalmanmasood · April 1, 2021, 7:46am

Oh I C… in this case, can we skip this validation since the gitlab server is limited to internal use only.

Many Thanks.

iwalker · April 1, 2021, 8:41am

No, you cannot skip validation with letsencrypt, even if it’s internal use only. Generate self-signed certificates, or use an internal CA for example. Letsencrypt requires a public DNS entry to be able to access your server, connect and verify to issue the certificate. And it needs to do this regularly, as certificates are only valid for 90 days, and cron jobs on the system will check regularly for this.

msalmanmasood · April 1, 2021, 8:42am

@iwalker so what are the recommendations for enabling SSL for internal use only?

iwalker · April 1, 2021, 9:24am

As per my previous message, the recommendation was generate self-signed certificate, or if you have an internal CA, then you can generate here also. If your internal domain that you are using is also the same as your commercial domain, eg: mydomain.com or whatever, then you can also purchase commercial certificate for that domain which would only require you to complete DV - domain validation - usually by receiving an email to verify you have control over it, and then once purchased, you can use that certificate on your internal server - even if the server is not publicly accessible.

There are plenty of articles on how to create self-signed certificates, or as well as configuring yourself a CA, be it on Linux or Windows. You can find these with google.

Then once generated, you put them in /etc/gitlab/ssl on your server with the files named as gitlab.mydomain.com.crt and gitlab.mydomain.com.key or whatever the URL is that you are using to access your server, and reconfigure gitlab and it’ll work with the certs, be it self-signed, CA generated or commercial purchased.

balonik · April 1, 2021, 1:37pm

@msalmanmasood just a small addition to what @iwalker already mentioned.
You can also manage Let’s encrypt certificate manually (outside of GitLab). When you do it yourself, you can choose what validation method you want to use.
Most Linux distributions already have some package to install and manage Let’s Encrypt certificates.

msalmanmasood · April 1, 2021, 1:49pm

So removed the letsencrypt SSL cert and key, applied the self-signed certificates by following the steps here [ 1] also commented out letsencrypt configurations and changed the parameter letsencrypt[‘enable’] = false and ran the gitlab-ctl reconfigure to apply the configurations as required.

Thanks @balonik and @iwalker for your continuous support throughout this effort here.

[1] - Using a Self Signed Certificate - #4 by ragnarok

CharlesDRoberts · April 2, 2021, 7:14pm

iwalker:

As per my previous message, the recommendation was generate self-signed certificate, or if you have an internal CA, then you can generate here also. If your internal domain that you are using is also the same as your commercial domain, eg: mydomain.com or whatever, then you can also purchase commercial certificate for that domain which would only require you to complete DV - domain validation - usually by receiving an email to verify you have control over it, and then once purchased, you can use that certificate on your internal server - even if the server is not publicly accessible.

There are plenty of articles on how to create self-signed certificates, or as well as configuring yourself a CA, be it on Linux or Windows. You can find these with google.

Then once generated, you put them in /etc/gitlab/ssl on your server with the files named as gitlab.mydomain.com.crt and gitlab.mydomain.com.key or whatever the URL is that you are using to access your server, and reconfigure gitlab and it’ll work with the certs, be it self-signed, CA generated or commercial purchased.

this Method is totally works for me thanks

CharlesDRoberts · April 2, 2021, 7:18pm

iwalker:

As per my previous message, the recommendation was generate self-signed certificate, or if you have an internal CA, then you can generate here also. If your internal domain that you are using is also the same as your commercial domain, eg: mydomain.com or whatever, then you can also purchase commercial certificate for that domain which would only require you to complete DV - domain validation - usually by receiving an email to verify you have control over it, and then once purchased, you can use that certificate on your internal server - even if the server is not publicly accessible.

There are plenty of articles on how to create self-signed certificates, or as well as configuring yourself a CA, be it on Linux or Windows. You can find these with google.

Then once generated, you put them in /etc/gitlab/ssl on your server with the files named as gitlab.mydomain.com.crt and gitlab.mydomain.com.key or whatever the URL is that you are using to access your server, and reconfigure gitlab and it’ll work with the certs, be it self-signed, CA generated or commercial purchased.

That work for me. Thanks