Let's Encrypt Fails with http_authorization line 6 error

How to Use GitLab Self-managed
1

Brand new installation of GitLab CE on stock Debian server. I have followed the directions here: SSL Configuration | GitLab and have read several other forum posts with similar descriptions. I have also tested the URL at https://letsdebug.net with no errors. Below are the changes I made to gitlab.rb and the output of the gitlab-ctl reconfigure command (truncated to comply with character limitations).

external_url 'https://my.domain.com'
letsencrypt['enable'] = true
letsencrypt['contact_emails'] = ['webmaster@my.domain.com'] # This should be an array of email addresses to add as contacts
letsencrypt['auto_renew'] = true
letsencrypt['auto_renew_hour'] = 17
letsencrypt['auto_renew_minute'] = 22 # Should be a number or cron expression, if specified.
letsencrypt['auto_renew_day_of_month'] = "*/20"

Starting Chef Infra Client, version 15.17.4
resolving cookbooks for run list: ["gitlab"]
Synchronizing Cookbooks:
  - gitlab (0.0.1)
  - package (0.1.0)
  - logrotate (0.1.0)
  - postgresql (0.1.0)
  - redis (0.1.0)
  - registry (0.1.0)
  - gitaly (0.1.0)
  - praefect (0.1.0)
  - gitlab-pages (0.1.0)
  - letsencrypt (0.1.0)
  - nginx (0.1.0)
  - runit (5.1.3)
  - acme (4.1.3)
  - crond (0.1.0)
  - monitoring (0.1.0)
  - gitlab-kas (0.1.0)
  - mattermost (0.1.0)
  - consul (0.1.0)
Installing Cookbook Gems:
Compiling Cookbooks...
...
...
...
Recipe: gitlab::nginx
  * directory[/var/opt/gitlab/nginx] action create (up to date)
  * directory[/var/opt/gitlab/nginx/conf] action create (up to date)
  * directory[/var/log/gitlab/nginx] action create (up to date)
  * link[/var/opt/gitlab/nginx/logs] action create (up to date)
  * template[/var/opt/gitlab/nginx/conf/gitlab-http.conf] action create (up to date)
  * template[/var/opt/gitlab/nginx/conf/gitlab-smartcard-http.conf] action delete (up to date)
  * template[/var/opt/gitlab/nginx/conf/gitlab-health.conf] action create (up to date)
  * template[/var/opt/gitlab/nginx/conf/gitlab-pages.conf] action delete (up to date)
  * template[/var/opt/gitlab/nginx/conf/gitlab-registry.conf] action create (up to date)
  * template[/var/opt/gitlab/nginx/conf/gitlab-mattermost-http.conf] action delete (up to date)
  * template[/var/opt/gitlab/nginx/conf/nginx-status.conf] action create (up to date)
  * consul_service[nginx] action delete
    * file[/var/opt/gitlab/consul/config.d/nginx-service.json] action delete (up to date)
     (up to date)
  * template[/var/opt/gitlab/nginx/conf/nginx.conf] action create (up to date)
Recipe: nginx::enable
  * service[nginx] action nothing (skipped due to action :nothing)
  * runit_service[nginx] action enable
    * ruby_block[restart_service] action nothing (skipped due to action :nothing)
    * ruby_block[restart_log_service] action nothing (skipped due to action :nothing)
    * ruby_block[reload_log_service] action nothing (skipped due to action :nothing)
    * directory[/opt/gitlab/sv/nginx] action create (up to date)
    * template[/opt/gitlab/sv/nginx/run] action create (up to date)
    * directory[/opt/gitlab/sv/nginx/log] action create (up to date)
    * directory[/opt/gitlab/sv/nginx/log/main] action create (up to date)
    * template[/opt/gitlab/sv/nginx/log/config] action create (up to date)
    * ruby_block[verify_chown_persisted_on_nginx] action nothing (skipped due to action :nothing)
    * link[/var/log/gitlab/nginx/config] action create (up to date)
    * template[/opt/gitlab/sv/nginx/log/run] action create (up to date)
    * directory[/opt/gitlab/sv/nginx/env] action create (up to date)
    * ruby_block[Delete unmanaged env files for nginx service] action run (skipped due to only_if)
    * template[/opt/gitlab/sv/nginx/check] action create (skipped due to only_if)
    * template[/opt/gitlab/sv/nginx/finish] action create (skipped due to only_if)
    * directory[/opt/gitlab/sv/nginx/control] action create (up to date)
    * link[/opt/gitlab/init/nginx] action create (up to date)
    * file[/opt/gitlab/sv/nginx/down] action nothing (skipped due to action :nothing)
    * directory[/opt/gitlab/service] action create (up to date)
    * link[/opt/gitlab/service/nginx] action create (up to date)
    * ruby_block[wait for nginx service socket] action run (skipped due to not_if)
     (up to date)
  * execute[reload nginx] action nothing (skipped due to action :nothing)
Recipe: gitlab::remote-syslog_disable
  * service[remote-syslog] action nothing (skipped due to action :nothing)
  * runit_service[remote-syslog] action disable
    * ruby_block[disable remote-syslog] action run (skipped due to only_if)
     (up to date)
Recipe: gitlab::storage-check_disable
  * service[storage-check] action nothing (skipped due to action :nothing)
  * runit_service[storage-check] action disable
    * ruby_block[disable storage-check] action run (skipped due to only_if)
     (up to date)
Recipe: gitlab-pages::disable
  * service[gitlab-pages] action nothing (skipped due to action :nothing)
  * runit_service[gitlab-pages] action disable
    * ruby_block[disable gitlab-pages] action run (skipped due to only_if)
     (up to date)
Recipe: registry::enable
  * directory[create /var/opt/gitlab/registry] action create (up to date)
  * account[Docker registry user and group] action create
    * group[Docker registry user and group] action create (up to date)
    * linux_user[Docker registry user and group] action create (up to date)
     (up to date)
  * directory[create /var/opt/gitlab/registry and set the owner] action create (up to date)
  * directory[create /var/log/gitlab/registry and set the owner] action create (up to date)
  * env_dir[/opt/gitlab/etc/registry/env] action create
    * directory[/opt/gitlab/etc/registry/env] action create (up to date)
    * file[/opt/gitlab/etc/registry/env/SSL_CERT_DIR] action create (up to date)
     (up to date)
  * directory[/var/opt/gitlab/gitlab-rails/shared/registry] action create (up to date)
  * file[/var/opt/gitlab/registry/gitlab-registry.crt] action create (up to date)
  * template[/var/opt/gitlab/registry/config.yml] action create (up to date)
  * service[registry] action nothing (skipped due to action :nothing)
  * runit_service[registry] action enable
    * ruby_block[restart_service] action nothing (skipped due to action :nothing)
    * ruby_block[restart_log_service] action nothing (skipped due to action :nothing)
    * ruby_block[reload_log_service] action nothing (skipped due to action :nothing)
    * directory[/opt/gitlab/sv/registry] action create (up to date)
    * template[/opt/gitlab/sv/registry/run] action create (up to date)
    * directory[/opt/gitlab/sv/registry/log] action create (up to date)
    * directory[/opt/gitlab/sv/registry/log/main] action create (up to date)
    * template[/opt/gitlab/sv/registry/log/config] action create (up to date)
    * ruby_block[verify_chown_persisted_on_registry] action nothing (skipped due to action :nothing)
    * link[/var/log/gitlab/registry/config] action create (up to date)
    * template[/opt/gitlab/sv/registry/log/run] action create (up to date)
    * directory[/opt/gitlab/sv/registry/env] action create (up to date)
    * ruby_block[Delete unmanaged env files for registry service] action run (skipped due to only_if)
    * template[/opt/gitlab/sv/registry/check] action create (skipped due to only_if)
    * template[/opt/gitlab/sv/registry/finish] action create (skipped due to only_if)
    * directory[/opt/gitlab/sv/registry/control] action create (up to date)
    * link[/opt/gitlab/init/registry] action create (up to date)
    * file[/opt/gitlab/sv/registry/down] action nothing (skipped due to action :nothing)
    * directory[/opt/gitlab/service] action create (up to date)
    * link[/opt/gitlab/service/registry] action create (up to date)
    * ruby_block[wait for registry service socket] action run (skipped due to not_if)
     (up to date)
  * version_file[Create version file for Registry] action create
    * file[/var/opt/gitlab/registry/VERSION] action create (up to date)
     (up to date)
Recipe: mattermost::disable
  * service[mattermost] action nothing (skipped due to action :nothing)
  * runit_service[mattermost] action disable
    * ruby_block[disable mattermost] action run (skipped due to only_if)
     (up to date)
Recipe: letsencrypt::enable
  * ruby_block[http external-url] action run (skipped due to only_if)
  * directory[/etc/gitlab/ssl] action create (up to date)
  * directory[/var/log/gitlab/lets-encrypt] action create (up to date)
  * acme_selfsigned[my.domain.com] action create
    * file[my.domain.com SSL selfsigned key] action create_if_missing (up to date)
    * file[my.domain.com SSL selfsigned crt] action create_if_missing (up to date)
    * file[my.domain.com SSL selfsigned chain] action create_if_missing (skipped due to not_if)
     (up to date)
Recipe: letsencrypt::http_authorization
  * letsencrypt_certificate[my.domain.com] action create
    * acme_certificate[staging] action create
      * file[my.domain.com SSL key] action create_if_missing (up to date)
      
      ================================================================================
      Error executing action `create` on resource 'acme_certificate[staging]'
      ================================================================================
      
      Acme::Client::Error::Timeout
      ----------------------------
      Acme::Client::Error::Timeout
      
      Cookbook Trace:
      ---------------
      /opt/gitlab/embedded/cookbooks/cache/cookbooks/acme/libraries/acme.rb:39:in `acme_client'
      /opt/gitlab/embedded/cookbooks/cache/cookbooks/acme/libraries/acme.rb:47:in `acme_order_certs_for'
      /opt/gitlab/embedded/cookbooks/cache/cookbooks/acme/resources/certificate.rb:87:in `block in class_from_file'
      
      Resource Declaration:
      ---------------------
      suppressed sensitive resource output
      
      Compiled Resource:
      ------------------
      suppressed sensitive resource output
      
      System Info:
      ------------
      chef_version=15.17.4
      platform=debian
      platform_version=11
      ruby=ruby 2.7.5p203 (2021-11-24 revision f69aeb8314) [x86_64-linux]
      program_name=/opt/gitlab/embedded/bin/chef-client
      executable=/opt/gitlab/embedded/bin/chef-client
      
    
    ================================================================================
    Error executing action `create` on resource 'letsencrypt_certificate[my.domain.com]'
    ================================================================================
    
    Acme::Client::Error::Timeout
    ----------------------------
    acme_certificate[staging] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/letsencrypt/resources/certificate.rb line 41) had an error: Acme::Client::Error::Timeout: Acme::Client::Error::Timeout
    
    Cookbook Trace:
    ---------------
    /opt/gitlab/embedded/cookbooks/cache/cookbooks/acme/libraries/acme.rb:39:in `acme_client'
    /opt/gitlab/embedded/cookbooks/cache/cookbooks/acme/libraries/acme.rb:47:in `acme_order_certs_for'
    /opt/gitlab/embedded/cookbooks/cache/cookbooks/acme/resources/certificate.rb:87:in `block in class_from_file'
    
    Resource Declaration:
    ---------------------
    # In /opt/gitlab/embedded/cookbooks/cache/cookbooks/letsencrypt/recipes/http_authorization.rb
    
      6: letsencrypt_certificate site do
      7:   crt node['gitlab']['nginx']['ssl_certificate']
      8:   key node['gitlab']['nginx']['ssl_certificate_key']
      9:   notifies :run, "execute[reload nginx]", :immediate
     10:   notifies :run, 'ruby_block[display_le_message]'
     11:   only_if { omnibus_helper.service_up?('nginx') }
     12: end
    
    Compiled Resource:
    ------------------
    # Declared in /opt/gitlab/embedded/cookbooks/cache/cookbooks/letsencrypt/recipes/http_authorization.rb:6:in `from_file'
    
    letsencrypt_certificate("my.domain.com") do
      action [:create]
      default_guard_interpreter :default
      declared_type :letsencrypt_certificate
      cookbook_name "letsencrypt"
      recipe_name "http_authorization"
      crt "/etc/gitlab/ssl/my.domain.com.crt"
      key "/etc/gitlab/ssl/my.domain.com.key"
      alt_names []
      cn "my.domain.com"
      only_if { #code block }
    end
    
    System Info:
    ------------
    chef_version=15.17.4
    platform=debian
    platform_version=11
    ruby=ruby 2.7.5p203 (2021-11-24 revision f69aeb8314) [x86_64-linux]
    program_name=/opt/gitlab/embedded/bin/chef-client
    executable=/opt/gitlab/embedded/bin/chef-client
    

Running handlers:
There was an error running gitlab-ctl reconfigure:

letsencrypt_certificate[my.domain.com] (letsencrypt::http_authorization line 6) had an error: Acme::Client::Error::Timeout: acme_certificate[staging] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/letsencrypt/resources/certificate.rb line 41) had an error: Acme::Client::Error::Timeout: Acme::Client::Error::Timeout

Running handlers complete
Chef Infra Client failed. 0 resources updated in 01 minutes 18 seconds
2

I had the same problem but I can’t fix it please any one face this error and fix it?

3

Hi did you find the solution?

4

I’m also running into this issue;

  1. running it all in a LXC container (Proxmox pct)
  2. Debian 12/bookworm
  3. gitlab-ce 16.4.0-ce.0

it seems as if nginx fails to respond when letsencrypt is trying to verify, the logs mentions removing the file just before the error is thrown so that could be a lead

[2023-09-23T21:27:36+00:00] INFO: file[obfuscated.domain.tld SSL selfsigned crt] group changed to 0
[2023-09-23T21:27:36+00:00] INFO: file[obfuscated.domain.tld SSL selfsigned crt] mode changed to 644

      - change mode from '' to '0644'
      - change owner from '' to 'root'
      - change group from '' to 'root'
    * file[obfuscated.domain.tld SSL selfsigned chain] action create_if_missing (skipped due to not_if)

[2023-09-23T21:27:36+00:00] INFO: acme_selfsigned[obfuscated.domain.tld] sending restart action to runit_service[nginx] (immediate)
Recipe: nginx::enable
  * runit_service[nginx] action restart (up to date)
Recipe: letsencrypt::http_authorization
  * letsencrypt_certificate[obfuscated.domain.tld] action create
    * acme_certificate[staging] action create
      * file[obfuscated.domain.tld SSL key] action nothing (skipped due to action :nothing)
      * file[obfuscated.domain.tld SSL key] action create_if_missing[2023-09-23T21:27:36+00:00] INFO: file[obfuscated.domain.tld SSL key] created file /etc/gitlab/ssl/obfuscated.hostname.local.key-staging

        - create new file /etc/gitlab/ssl/obfuscated.hostname.local.key-staging[2023-09-23T21:27:36+00:00] INFO: file[obfuscated.domain.tld SSL key] updated file contents /etc/gitlab/ssl/obfuscated.hostname.local.key-staging

        - update content in file /etc/gitlab/ssl/obfuscated.hostname.local.key-staging from none to ca219b
        - suppressed sensitive resource[2023-09-23T21:27:36+00:00] INFO: file[obfuscated.domain.tld SSL key] owner changed to 0
[2023-09-23T21:27:36+00:00] INFO: file[obfuscated.domain.tld SSL key] group changed to 0
[2023-09-23T21:27:36+00:00] INFO: file[obfuscated.domain.tld SSL key] mode changed to 400

        - change mode from '' to '0400'
        - change owner from '' to 'root'
        - change group from '' to 'root'
      * directory[/var/opt/gitlab/nginx/www/.well-known/acme-challenge] action nothing (skipped due to action :nothing)
      * directory[/var/opt/gitlab/nginx/www/.well-known/acme-challenge] action create (up to date)
      * file[/var/opt/gitlab/nginx/www/.well-known/acme-challenge/wtHNMmS2Fk7YYM3iUfr-io-a-89zYTcrWlr2fGo07V4] action nothing (skipped due to action :nothing)
      * file[/var/opt/gitlab/nginx/www/.well-known/acme-challenge/wtHNMmS2Fk7YYM3iUfr-io-a-89zYTcrWlr2fGo07V4] action create[2023-09-23T21:27:40+00:00] INFO: file[/var/opt/gitlab/nginx/www/.well-known/acme-challenge/wtHNMmS2Fk7YYM3iUfr-io-a-89zYTcrWlr2fGo07V4] created file /var/opt/gitlab/nginx/www/.well-known/acme-challenge/wtHNMmS2Fk7YYM3iUfr-io-a-89zYTcrWlr2fGo07V4

        - create new file /var/opt/gitlab/nginx/www/.well-known/acme-challenge/wtHNMmS2Fk7YYM3iUfr-io-a-89zYTcrWlr2fGo07V4[2023-09-23T21:27:40+00:00] INFO: file[/var/opt/gitlab/nginx/www/.well-known/acme-challenge/wtHNMmS2Fk7YYM3iUfr-io-a-89zYTcrWlr2fGo07V4] updated file contents /var/opt/gitlab/nginx/www/.well-known/acme-challenge/wtHNMmS2Fk7YYM3iUfr-io-a-89zYTcrWlr2fGo07V4

        - update content in file /var/opt/gitlab/nginx/www/.well-known/acme-challenge/wtHNMmS2Fk7YYM3iUfr-io-a-89zYTcrWlr2fGo07V4 from none to 4ddbdf
        --- /var/opt/gitlab/nginx/www/.well-known/acme-challenge/wtHNMmS2Fk7YYM3iUfr-io-a-89zYTcrWlr2fGo07V4    2023-09-23 21:27:40.017686259 +0000
        +++ /var/opt/gitlab/nginx/www/.well-known/acme-challenge/.chef-wtHNMmS2Fk7YYM3iUfr-io-a-89zYTcrWlr2fGo07V420230923-6940-ptjj8p  2023-09-23 21:27:40.017686259 +0000
        @@ -1 +1,2 @@
        +wtHNMmS2Fk7YYM3iUfr-io-a-89zYTcrWlr2fGo07V4.cO1s7JBuCL47oXc8bm1tLtzBjokY5zrFL-zAs75bYEs[2023-09-23T21:27:40+00:00] INFO: file[/var/opt/gitlab/nginx/www/.well-known/acme-challenge/wtHNMmS2Fk7YYM3iUfr-io-a-89zYTcrWlr2fGo07V4] owner changed to 0
[2023-09-23T21:27:40+00:00] INFO: file[/var/opt/gitlab/nginx/www/.well-known/acme-challenge/wtHNMmS2Fk7YYM3iUfr-io-a-89zYTcrWlr2fGo07V4] group changed to 0
[2023-09-23T21:27:40+00:00] INFO: file[/var/opt/gitlab/nginx/www/.well-known/acme-challenge/wtHNMmS2Fk7YYM3iUfr-io-a-89zYTcrWlr2fGo07V4] mode changed to 644

        - change mode from '' to '0644'
        - change owner from '' to 'root'
        - change group from '' to 'root'
      * file[/var/opt/gitlab/nginx/www/.well-known/acme-challenge/wtHNMmS2Fk7YYM3iUfr-io-a-89zYTcrWlr2fGo07V4] action delete[2023-09-23T21:27:42+00:00] INFO: file[/var/opt/gitlab/nginx/www/.well-known/acme-challenge/wtHNMmS2Fk7YYM3iUfr-io-a-89zYTcrWlr2fGo07V4] deleted file at /var/opt/gitlab/nginx/www/.well-known/acme-challenge/wtHNMmS2Fk7YYM3iUfr-io-a-89zYTcrWlr2fGo07V4

        - delete file /var/opt/gitlab/nginx/www/.well-known/acme-challenge/wtHNMmS2Fk7YYM3iUfr-io-a-89zYTcrWlr2fGo07V4
      * ruby_block[create certificate for obfuscated.domain.tld] action run

        ================================================================================
        Error executing action `run` on resource 'ruby_block[create certificate for obfuscated.domain.tld]'
        ================================================================================

        RuntimeError
        ------------
        [obfuscated.domain.tld] Validation failed, unable to request certificate, Errors: [{url: https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/8464019324/Yu-z3g, status: invalid, error: {"type"=>"urn:ietf:params:acme:error:connection", "detail"=>"31.208.34.94: Fetching http://obfuscated.domain.tld/.well-known/acme-challenge/wtHNMmS2Fk7YYM3iUfr-io-a-89zYTcrWlr2fGo07V4: Error getting validation data", "status"=>400}} ]

        Cookbook Trace: (most recent call first)
        ----------------------------------------
        /opt/gitlab/embedded/cookbooks/cache/cookbooks/acme/resources/certificate.rb:117:in `block (3 levels) in class_from_file'
        /opt/gitlab/embedded/cookbooks/cache/cookbooks/acme/resources/certificate.rb:110:in `block in class_from_file'
        /opt/gitlab/embedded/cookbooks/cache/cookbooks/letsencrypt/resources/certificate.rb:43:in `block in class_from_file'

        Resource Declaration:
        ---------------------
        # In /opt/gitlab/embedded/cookbooks/cache/cookbooks/acme/resources/certificate.rb

        110:     ruby_block "create certificate for #{new_resource.cn}" do
        111:       block do
        112:         unless (all_validations.map { |authz| authz.status == 'valid' }).all?
        113:           errors = all_validations.select { |authz| authz.status != 'valid' }.map do |authz|
        114:             "{url: #{authz.url}, status: #{authz.status}, error: #{authz.error}} "
        115:           end.reduce(:+)
        116:
        117:           raise "[#{new_resource.cn}] Validation failed, unable to request certificate, Errors: [#{errors}]"
        118:         end
        119:
        120:         begin
        121:           newcert = acme_cert(order, new_resource.cn, mykey, new_resource.alt_names)
        122:         rescue Acme::Client::Error => e
        123:           raise "[#{new_resource.cn}] Certificate request failed: #{e.message}"
        124:         else
        125:           Chef::Resource::File.new("#{new_resource.cn} SSL new crt", run_context).tap do |f|
        126:             f.path    new_resource.crt
        127:             f.owner   new_resource.owner
        128:             f.group   new_resource.group
        129:             f.content newcert
        130:             f.mode    00644
        131:           end.run_action :create
        132:         end
        133:       end
        134:     end
        135:   end

        Compiled Resource:
        ------------------
        # Declared in /opt/gitlab/embedded/cookbooks/cache/cookbooks/acme/resources/certificate.rb:110:in `block in class_from_file'

        ruby_block("create certificate for obfuscated.domain.tld") do
          action [:run]
          default_guard_interpreter :default
          declared_type :ruby_block
          cookbook_name "letsencrypt"
          recipe_name "http_authorization"
          block #<Proc:0x00007f8f10502f98 /opt/gitlab/embedded/cookbooks/cache/cookbooks/acme/resources/certificate.rb:111>
        end