LetsEncrypt certificates fail in domain validation

I was up until now getting some LE certificates manually renewed using certbot but decided to move to automatically managed certificates in gitlab 11.0.1.

However, after setting up the proper variables in gitlab.rb , I am getting during a gitlab-ctl reconfigure:

Recipe: letsencrypt::http_authorization
  * letsencrypt_certificate[gitlab.linki.tools] action create
    * acme_certificate[staging] action create
      * file[gitlab.linki.tools SSL key] action create_if_missing (up to date)
      * directory[/var/opt/gitlab/nginx/www/.well-known/acme-challenge] action create (up to date)
      * file[/var/opt/gitlab/nginx/www/.well-known/acme-challenge/RrXdMD2cTteB4BZ75SQDPeKMKVLyA5R6aq84xmcBw7I] action create
        - create new file /var/opt/gitlab/nginx/www/.well-known/acme-challenge/RrXdMD2cTteB4BZ75SQDPeKMKVLyA5R6aq84xmcBw7I
        - update content in file /var/opt/gitlab/nginx/www/.well-known/acme-challenge/RrXdMD2cTteB4BZ75SQDPeKMKVLyA5R6aq84xmcBw7I from none to d0a818
        --- /var/opt/gitlab/nginx/www/.well-known/acme-challenge/RrXdMD2cTteB4BZ75SQDPeKMKVLyA5R6aq84xmcBw7I       2018-08-10 10:19:50.504372251 +0200
        +++ /var/opt/gitlab/nginx/www/.well-known/acme-challenge/.chef-RrXdMD2cTteB4BZ75SQDPeKMKVLyA5R6aq84xmcBw7I20180810-4569-13jvrfq       2018-08-10 10:19:50.504372251 +0200
        @@ -1 +1,2 @@
        +RrXdMD2cTteB4BZ75SQDPeKMKVLyA5R6aq84xmcBw7I.ZMs-47uotHRPS6SEnUAnglLQl2aJN_jnKJk2kzX4ZfA
        - change mode from '' to '0644'
        - change owner from '' to 'root'
        - change group from '' to 'root'
      
      ================================================================================
      Error executing action `create` on resource 'acme_certificate[staging]'
      ================================================================================
      
      RuntimeError
      ------------
      [gitlab.linki.tools] Validation failed for domain gitlab.linki.tools

      Cookbook Trace:
      ---------------
      /opt/gitlab/embedded/cookbooks/cache/cookbooks/acme/providers/certificate.rb:93:in `block (2 levels) in class_from_file'
      /opt/gitlab/embedded/cookbooks/cache/cookbooks/acme/providers/certificate.rb:68:in `map'
      /opt/gitlab/embedded/cookbooks/cache/cookbooks/acme/providers/certificate.rb:68:in `block in class_from_file'
      
      Resource Declaration:
      ---------------------
      suppressed sensitive resource output
      
      Compiled Resource:
      ------------------
      suppressed sensitive resource output
      
      System Info:
      ------------
      chef_version=13.6.4
      platform=ubuntu
      platform_version=16.04
      ruby=ruby 2.4.4p296 (2018-03-28 revision 63013) [x86_64-linux]
      program_name=/opt/gitlab/embedded/bin/chef-client
      executable=/opt/gitlab/embedded/bin/chef-client
      
    
    ================================================================================
    Error executing action `create` on resource 'letsencrypt_certificate[gitlab.linki.tools]'
    ================================================================================
    
    RuntimeError
    ------------
    acme_certificate[staging] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/letsencrypt/resources/certificate.rb line 20) had an error: RuntimeError: [gitlab.linki.tools] Validation failed for domain gitlab.linki.tools
    
    Cookbook Trace:
    ---------------
    /opt/gitlab/embedded/cookbooks/cache/cookbooks/acme/providers/certificate.rb:93:in `block (2 levels) in class_from_file'
    /opt/gitlab/embedded/cookbooks/cache/cookbooks/acme/providers/certificate.rb:68:in `map'
    /opt/gitlab/embedded/cookbooks/cache/cookbooks/acme/providers/certificate.rb:68:in `block in class_from_file'

    Resource Declaration:
    ---------------------
    # In /opt/gitlab/embedded/cookbooks/cache/cookbooks/letsencrypt/recipes/http_authorization.rb
    
      3: letsencrypt_certificate site do
      4:   fullchain node['gitlab']['nginx']['ssl_certificate']
      5:   key node['gitlab']['nginx']['ssl_certificate_key']
      6:   notifies :run, "execute[reload nginx]", :immediate
      7:   notifies :run, 'ruby_block[display_le_message]'
      8: end
    
    Compiled Resource:
    ------------------
    # Declared in /opt/gitlab/embedded/cookbooks/cache/cookbooks/letsencrypt/recipes/http_authorization.rb:3:in `from_file'
    
    letsencrypt_certificate("gitlab.linki.tools") do
      action [:create]
      updated true
      updated_by_last_action true
      default_guard_interpreter :default
      declared_type :letsencrypt_certificate
      cookbook_name "letsencrypt"
      recipe_name "http_authorization"
      fullchain "/etc/letsencrypt/live/gitlab.linki.tools/fullchain.pem"
      key "/etc/letsencrypt/live/gitlab.linki.tools/privkey.pem"
      alt_names ["registry.linki.tools"]
      cn "gitlab.linki.tools"
    end
    
    System Info:
    ------------
    chef_version=13.6.4
    platform=ubuntu
    platform_version=16.04
    ruby=ruby 2.4.4p296 (2018-03-28 revision 63013) [x86_64-linux]
    program_name=/opt/gitlab/embedded/bin/chef-client
    executable=/opt/gitlab/embedded/bin/chef-client

Running handlers:
There was an error running gitlab-ctl reconfigure:

letsencrypt_certificate[gitlab.linki.tools] (letsencrypt::http_authorization line 3) had an error: RuntimeError: acme_certificate[staging] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/letsencrypt/resources/certificate.rb line 20) had an error: RuntimeError: [gitlab.linki.tools] Validation failed for domain gitlab.linki.tools

Running handlers complete
Chef Client failed. 3 resources updated in 12 seconds

Why can’t the validation be completed for my gitlab.linki.tools domain? Can I find more information somewhere else?

Okay, I solved this myself by removing all nginx ssl path configurations and adding:

nginx['redirect_http_to_https'] = true
nginx['redirect_http_to_https_port'] = 80
4 Likes

Hello pmatos, I have exactly the same problem. Which nginx ssl path configurations did you remove to get it working?

I just had the same issue but with mattermost, I was able to fix it by commenting out

mattermost_nginx['redirect_http_to_https'] = true

and running reconfigure

2 Likes

I have the same question! What specific SSL configurations did you comment out?

I had the same issue, but, unfortunately, pmatos’s solution didn’t work for me. I’m using Gitlab from docker image gitlab/gitlab-ce:11.3.3-ce.0.

In my case the main reason of the error Validation failed for domain was the fact that during validation process certbot put special file to the directory /var/opt/gitlab/nginx/www/.well-known/acme-challenge/, in my case it was something like:

/var/opt/gitlab/nginx/www/.well-known/acme-challenge/V0kLCyM7gI7q5LNxir0iIL2dSM0RimDgx91MtuBgbf9

Then Letsencrypt trying to request the file http://example.com/.well-known/acme-challenge/V0kLCyM7gI7q5LNxir0iIL2dSM0RimDgx91MtuBgbf9, but in my case this file wasn’t accessible via http!

To solve this issue I added following line to the /etc/gitlab/gitlab.rb:

nginx['custom_gitlab_server_config'] = "location /.well-known/acme-challenge/ {\n root /var/opt/gitlab/nginx/www/; \n}\n"

This makes files in the directory /var/opt/gitlab/nginx/www/.well-known/acme-challenge/ accessible via http.

Also I had to temporary turn off https:

letsencrypt['enable'] = false

(with enabled letsecrypt my docker container couldn’t start, when container started without letsencrypt support, I turned this option back).

Also I removed all old certificates from /etc/gitlab/ssl/.

Then I started docker container and run:

gitlab-ctl renew-le-certs

New certificates successfully appeared in /etc/gitlab/ssl/. Then I run:

gitlab-ctl reconfigure

Eventually Gitlab started succssfully!

9 Likes

Thanks romka. This line fixed it for me too:

nginx['custom_gitlab_server_config'] = "location /.well-known/acme-challenge/ {\n root /var/opt/gitlab/nginx/www/; \n}\n"

I had the redirect_http_to_https settings enabled already and those alone didn’t fix it.

2 Likes

I can’t get any of this to work. I’m not sure but does my subdomain have to be available in DNS for letsencrypt to be able to verify it?

I’m pretty sure this is the dumbest of all solutions, but I simply forgot to turn off apache2 since it is blocking the port 80.

solved the issue for me,

letsencrypt['enable'] = true
letsencrypt['auto_renew'] = true
letsencrypt['auto_renew_hour'] = 0
letsencrypt['auto_renew_minute'] = 30
letsencrypt['auto_renew_day_of_month'] = "*/4"
nginx['custom_gitlab_server_config'] = "location /.well-known/acme-challenge/ {\n root /var/opt/gitlab/nginx/www/; \n}\n"
1 Like

Thx a lot mate :slight_smile:

Related to the port 80:

  • HTTP-01:
    “Let’s Encrypt gives a token to your ACME client, and your ACME client puts a file on your web server at http://<YOUR_DOMAIN>/.well-known/acme-challenge/<TOKEN>
  • DNS-01:
    “This challenge asks you to prove that you control the DNS for your domain name by putting a specific value in a TXT record under that domain name.”

About the challenge types:

ACME client:

1 Like

Yet regarding the “DNS challenge” (DNS-01), I found this another topic: Validation Of Let's Encrypt - Help - Let's Encrypt Community Support

Needed to do this when upgrading to v12.0.0

It won’t validate for me. I’ve added a test.html file to the /var/opt/gitlab/nginx/www/.well-known/acme-challenge/ folder and can access it via:
http://gitlab.notarealdomain.comcom/.well-known/acme-challenge/test.html

I can’t access the random file letsencypt needs to access though because it gets deleted by the system automatically. It looks like it gets deleted before the validate runs? Would that be the cause? Here is the output:

  • file[/var/opt/gitlab/nginx/www/.well-known/acme-challenge/_KXmRB3XKk-aIs-Z98NZc_q2nNhLML-HDvADsPIn3vU] action delete
    - delete file /var/opt/gitlab/nginx/www/.well-known/acme-challenge/_KXmRB3XKk-aIs-Z98NZc_q2nNhLML-HDvADsPIn3vU <— file gets delete there
    • ruby_block[create certificate for gitlab.notarealdomain.comcom] action run
      Error executing action run on resource ‘ruby_block[create certificate for gitlab.notarealdomain.comcom]’

      RuntimeError

      [gitlab.notarealdomain.comcom] Validation failed, unable to request certificate

Got a quite similar error.
It was my mistake, tho.
I’ve forgotten to change the external URL from http to httpS.

that’s precious advice. solved my issue after 3 hour struggle.

if anyone still has problem with letsencrypt certificates after gitlab update, I resolved the issue by including the custom_gitlab_server_config posted above AND removing current certs from /etc/gitlab/ssl/. gitlab-ctl reconfigure did run successfully after :grinning:

1 Like

I faced the similar problem/error.

@romka 's answer helped me.

Here are the detailed steps : https://gitlab.com/gitlab-org/omnibus-gitlab/issues/4900#note_292031191

Roman! You’re helped me so much! I tried to solve it as long as it possible :frowning:
Thanks !