Let`s Encrypt Cannot Validate Certificate

General
,
1

I just read that letsencrypt in does not work with custom port for certificate validation. However, port 80 and 443 already used by another application. I have replace port 80 to port 10080 and port 443 to port 10443. After enable Let`s Encrypt in gitlab.rb, I encounter the “unable request certificate”. Below are the “gitlab-ctl reconfigure” logs.

Recipe: letsencrypt::enable
          * ruby_block[http external-url] action run (skipped due to only_if)
          * directory[/etc/gitlab/ssl] action create (up to date)
          * acme_selfsigned[[example.com] action create
            * file[[example.com SSL selfsigned key] action create_if_missing (up to date)
            * file[[example.com SSL selfsigned crt] action create_if_missing (up to date)
            * file[[example.com SSL selfsigned chain] action create_if_missing (skipped due to not_if)
             (up to date)
        Recipe: letsencrypt::http_authorization
          * letsencrypt_certificate[[example.com] action create
            * acme_certificate[staging] action create
              * file[[example.com SSL key] action create_if_missing (up to date)
              * directory[/var/opt/gitlab/nginx/www/.well-known/acme-challenge] action create (up to date)
              * file[/var/opt/gitlab/nginx/www/.well-known/acme-challenge/n0JqGJrVIn_yJZoFupWGHGC5-G2lYgr0YtRxC1mBOrU] action create
                - create new file /var/opt/gitlab/nginx/www/.well-known/acme-challenge/n0JqGJrVIn_yJZoFupWGHGC5-G2lYgr0YtRxC1mBOrU
                - update content in file /var/opt/gitlab/nginx/www/.well-known/acme-challenge/n0JqGJrVIn_yJZoFupWGHGC5-G2lYgr0YtRxC1mBOrU from none to c2cd13
                --- /var/opt/gitlab/nginx/www/.well-known/acme-challenge/n0JqGJrVIn_yJZoFupWGHGC5-G2lYgr0YtRxC1mBOrU    2020-11-24 14:31:57.841252396 +0000
                +++ /var/opt/gitlab/nginx/www/.well-known/acme-challenge/.chef-n0JqGJrVIn_yJZoFupWGHGC5-G2lYgr0YtRxC1mBOrU20201124-109040-8rdph4      2020-11-24 14:31:57.838252413 +0000
                @@ -1 +1,2 @@
                +n0JqGJrVIn_yJZoFupWGHGC5-G2lYgr0YtRxC1mBOrU.yIZAkFjOZWsQ6tbADwvfYpTyyriQBNrVs1r32FvAJkY
                - change mode from '' to '0644'
                - change owner from '' to 'root'
                - change group from '' to 'root'
              * file[[example.com SSL key] action nothing (skipped due to action :nothing)
              * directory[/var/opt/gitlab/nginx/www/.well-known/acme-challenge] action nothing (skipped due to action :nothing)
              * file[/var/opt/gitlab/nginx/www/.well-known/acme-challenge/n0JqGJrVIn_yJZoFupWGHGC5-G2lYgr0YtRxC1mBOrU] action nothing (skipped due to action :nothing)
              * file[/var/opt/gitlab/nginx/www/.well-known/acme-challenge/n0JqGJrVIn_yJZoFupWGHGC5-G2lYgr0YtRxC1mBOrU] action delete
                - delete file /var/opt/gitlab/nginx/www/.well-known/acme-challenge/n0JqGJrVIn_yJZoFupWGHGC5-G2lYgr0YtRxC1mBOrU
              * ruby_block[create certificate for [example.com] action run

                ================================================================================
                Error executing action `run` on resource 'ruby_block[create certificate for [example.com]'
                ================================================================================

                RuntimeError
                ------------
                [[example.com] Validation failed, unable to request certificate

                Cookbook Trace:
                ---------------
                /opt/gitlab/embedded/cookbooks/cache/cookbooks/acme/resources/certificate.rb:111:in `block (3 levels) in class_from_file'

                Resource Declaration:
                ---------------------
                # In /opt/gitlab/embedded/cookbooks/cache/cookbooks/acme/resources/certificate.rb

                108:     ruby_block "create certificate for #{new_resource.cn}" do # ~FC014
                109:       block do
                110:         unless (all_validations.map { |authz| authz.status == 'valid' }).all?
                111:           fail "[#{new_resource.cn}] Validation failed, unable to request certificate"
                112:         end
                113:
                114:         begin
                115:           newcert = acme_cert(order, new_resource.cn, mykey, new_resource.alt_names)
                116:         rescue Acme::Client::Error => e
                117:           fail "[#{new_resource.cn}] Certificate request failed: #{e.message}"
                118:         else
                119:           Chef::Resource::File.new("#{new_resource.cn} SSL new crt", run_context).tap do |f|
                120:             f.path    new_resource.crt
                121:             f.owner   new_resource.owner
                122:             f.group   new_resource.group
                123:             f.content newcert
                124:             f.mode    00644
                125:           end.run_action :create
                126:         end
                127:       end
                128:     end

                Compiled Resource:
                ------------------
                # Declared in /opt/gitlab/embedded/cookbooks/cache/cookbooks/acme/resources/certificate.rb:108:in `block in class_from_file'

                ruby_block("create certificate for [example.com") do
                  action [:run]
                  default_guard_interpreter :default
                  declared_type :ruby_block
                  cookbook_name "letsencrypt"
                  block #<Proc:0x0000000004d17c58 /opt/gitlab/embedded/cookbooks/cache/cookbooks/acme/resources/certificate.rb:109>
                  block_name "create certificate for [example.com"
                end

                System Info:
                ------------
                chef_version=15.14.0
                platform=ubuntu
                platform_version=16.04
                ruby=ruby 2.7.2p137 (2020-10-01 revision 5445e04352) [x86_64-linux]
                program_name=/opt/gitlab/embedded/bin/chef-client
                executable=/opt/gitlab/embedded/bin/chef-client


              ================================================================================
              Error executing action `create` on resource 'acme_certificate[staging]'
              ================================================================================

              RuntimeError
              ------------
              ruby_block[create certificate for [example.com] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/acme/resources/certificate.rb line 108) had an error: RuntimeError: [[example.com] Validation failed, unable to request certificate

              Cookbook Trace:
              ---------------
              /opt/gitlab/embedded/cookbooks/cache/cookbooks/acme/resources/certificate.rb:111:in `block (3 levels) in class_from_file'

              Resource Declaration:
              ---------------------
              suppressed sensitive resource output

              Compiled Resource:
              ------------------
              suppressed sensitive resource output

              System Info:
              ------------
              chef_version=15.14.0
              platform=ubuntu
              platform_version=16.04
              ruby=ruby 2.7.2p137 (2020-10-01 revision 5445e04352) [x86_64-linux]
              program_name=/opt/gitlab/embedded/bin/chef-client
              executable=/opt/gitlab/embedded/bin/chef-client


            ================================================================================
            Error executing action `create` on resource 'letsencrypt_certificate[[example.com]'
            ================================================================================

            RuntimeError
            ------------
            acme_certificate[staging] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/letsencrypt/resources/certificate.rb line 25) had an error: RuntimeError: ruby_block[create certificate for [example.com] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/acme/resources/certificate.rb line 108) had an error: RuntimeError: [[example.com] Validation failed, unable to request certificate

            Cookbook Trace:
            ---------------
            /opt/gitlab/embedded/cookbooks/cache/cookbooks/acme/resources/certificate.rb:111:in `block (3 levels) in class_from_file'

            Resource Declaration:
            ---------------------
            # In /opt/gitlab/embedded/cookbooks/cache/cookbooks/letsencrypt/recipes/http_authorization.rb

              5: letsencrypt_certificate site do
              6:   crt node['gitlab']['nginx']['ssl_certificate']
              7:   key node['gitlab']['nginx']['ssl_certificate_key']
              8:   notifies :run, "execute[reload nginx]", :immediate
              9:   notifies :run, 'ruby_block[display_le_message]'
             10:   only_if { omnibus_helper.service_up?('nginx') }
             11: end

            Compiled Resource:
            ------------------
            # Declared in /opt/gitlab/embedded/cookbooks/cache/cookbooks/letsencrypt/recipes/http_authorization.rb:5:in `from_file'

            letsencrypt_certificate("[example.com") do
              action [:create]
              updated true
              updated_by_last_action true
              default_guard_interpreter :default
              declared_type :letsencrypt_certificate
              cookbook_name "letsencrypt"
              recipe_name "http_authorization"
              crt "/etc/gitlab/ssl/[example.com.crt"
              key "/etc/gitlab/ssl/[example.com.key"
              alt_names []
              cn "[example.com"
              only_if { #code block }
            end

            System Info:
            ------------
            chef_version=15.14.0
            platform=ubuntu
            platform_version=16.04
            ruby=ruby 2.7.2p137 (2020-10-01 revision 5445e04352) [x86_64-linux]
            program_name=/opt/gitlab/embedded/bin/chef-client
            executable=/opt/gitlab/embedded/bin/chef-client

        Recipe: nginx::enable
          * runit_service[nginx] action restart (up to date)

        Running handlers:
        There was an error running gitlab-ctl reconfigure:

        letsencrypt_certificate[[example.com] (letsencrypt::http_authorization line 5) had an error: RuntimeError: acme_certificate[staging] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/letsencrypt/resources/certificate.rb line 25) had an error: RuntimeError: ruby_block[create certificate for [example.com] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/acme/resources/certificate.rb line 108) had an error: RuntimeError: [example.com] Validation failed, unable to request certificate


        Deprecations:
        * sidekiq_cluster['experimental_queue_selector'] has been deprecated since 13.6 and will be removed in 14.0. The experimental_queue_selector option is now called queue_selector.

        Running handlers complete
        Chef Infra Client failed. 5 resources updated in 18 seconds

        Deprecations:
        * sidekiq_cluster['experimental_queue_selector'] has been deprecated since 13.6 and will be removed in 14.0. The experimental_queue_selector option is now called queue_selector.

Is there another open source SSL certificate provider I can use other than letsencrypt.

2

I have the same problem, in the same step, trying to run the sample playbook with gitlab/gitlab-ee:latest image

3

I try to deploy it in Google Cloud Engine using Free Tier account… using the standard port 80 and 443. It work perfectly fine. No need to configure the gitlab.rb file. The problem started when I deploy it using custom port. Still figuring out to solve the SSL issue.

4

You might be able to include a rule on the conf that runs on the default port to direct the validation URI (
.well-known/acme-challenge) to the indicated directory (/var/opt/gitlab/nginx/www/.well-known/acme-challenge). If your current site also uses letsencrypt, you can configure the www-root location for that in the letsencrypt configuration.

That way your other configuration can answer the validation request. Not sure if other validation options (like dns challenge) can be setup in gitlab, also, you’ll probably want to automate that as well, which will trigger some more problems.

1 Like
5

My colleague and I manage to integrate Let`s Encrypt using certbot and external Nginx. We generate the certificate via certbot tool installed in our host server. Then we create Nginx reverse proxy configuration for our domain. Below is the configuration we created to redirect HTTP request to HTTPS with generated certificates.

server {
        listen 80;
        server_name SERVER_NAME;
        return 301 https://$server_name$request_uri;
}

server {
        listen 443 ssl http2;
        server_name SERVER_NAME;

        ssl_certificate /etc/letsencrypt/live/DOMAIN_NAME/fullchain.pem;
        ssl_certificate_key /etc/letsencrypt/live/DOMAIN_NAME/privkey.pem;
        ssl_prefer_server_ciphers on;
        ssl_dhparam /etc/ssl/certs/dhparam.pem;
        ssl_ciphers EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH;

        location / {
                proxy_set_header Host                   $host;
                proxy_set_header X-Real-IP              $remote_addr;
                proxy_set_header X-Forwarded-For        $proxy_add_x_forwarded_for;
                proxy_pass https://0.0.0.0:PORT; # Listen to from all IP addresses with Gitlab docker assigned port
        }
}