Letsencrypt failed on gitlab-ctl reconfigure help (14.5.2-ce.0)

General
1

Hello,

Been having some issues with my gitlab self hosted server.
Seem to occur every time I turn off my server and don’t turn it on for a week or so.

So I’ve had it all working but when I turned it on this morning it’s not working anymore (Had the server working for over 4 months and being turned off and on)

The issue is this

There was an error running gitlab-ctl reconfigure:

letsencrypt_certificate[git.servername.com] (letsencrypt::http_authorization line 6) had an error: RuntimeError: acme_certificate[staging] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/letsencrypt/resources/certificate.rb line 41) had an error: RuntimeError: ruby_block[create certificate for git.servername.com] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/acme/resources/certificate.rb line 108) had an error: RuntimeError: [git.servername.com] Validation failed, unable to request certificate, Errors: [{url: https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/1225710228/hgTISA, status: invalid, error: {"type"=>"urn:ietf:params:acme:error:connection", "detail"=>"Fetching http://git.servername.com/.well-known/acme-challenge/OTnoL50zGUIbm5cSGgQOBaQMJQq_8NREmKIa0oSyvQA: Timeout during connect (likely firewall problem)", "status"=>400}} ]

As it mentions in the log above - “ikely firewall problem” and probably is but I am really stumped as I’ve ported everything I needed to on the server

[ 1] OpenSSH                    ALLOW IN    Anywhere
[ 2] 80/tcp                     ALLOW IN    Anywhere
[ 3] 443/tcp                    ALLOW IN    Anywhere
[ 4] Postfix                    ALLOW IN    Anywhere
[ 5] 8082                       ALLOW IN    Anywhere
[ 6] 80                         ALLOW IN    Anywhere
[ 7] 443                        ALLOW IN    Anywhere
[ 8] 5000/tcp                   ALLOW IN    Anywhere
[ 9] 22/tcp                     ALLOW IN    Anywhere
[10] Nginx Full                 ALLOW IN    Anywhere
[11] 8888                       ALLOW IN    Anywhere
[12] OpenSSH (v6)               ALLOW IN    Anywhere (v6)
[13] 80/tcp (v6)                ALLOW IN    Anywhere (v6)
[14] 443/tcp (v6)               ALLOW IN    Anywhere (v6)
[15] Postfix (v6)               ALLOW IN    Anywhere (v6)
[16] 8082 (v6)                  ALLOW IN    Anywhere (v6)
[17] 80 (v6)                    ALLOW IN    Anywhere (v6)
[18] 443 (v6)                   ALLOW IN    Anywhere (v6)
[19] 5000/tcp (v6)              ALLOW IN    Anywhere (v6)
[20] 22/tcp (v6)                ALLOW IN    Anywhere (v6)
[21] Nginx Full (v6)            ALLOW IN    Anywhere (v6)
[22] 8888 (v6)                  ALLOW IN    Anywhere (v6)

I have also ported on my router (Which is a Sky router)

ports
ports594×927 65.1 KB

If anyone knows what else I can do please say

2

Here is more info

        Error executing action `run` on resource 'ruby_block[create certificate for git.rubberbandgames.com]'
        ================================================================================

        RuntimeError
        ------------
        [git.rubberbandgames.com] Validation failed, unable to request certificate, Errors: [{url: https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/1226773828/Z2irTg, status: invalid, error: {"type"=>"urn:ietf:params:acme:error:connection", "detail"=>"Fetching http://git.rubberbandgames.com/.well-known/acme-challenge/E8NuYYNIsIvXUPyEfy64L5fwT2cZbCr92m8CXgJyUC0: Timeout during connect (likely firewall problem)", "status"=>400}} ]

        Cookbook Trace:
        ---------------
        /opt/gitlab/embedded/cookbooks/cache/cookbooks/acme/resources/certificate.rb:117:in `block (3 levels) in class_from_file'

        Resource Declaration:
        ---------------------
        # In /opt/gitlab/embedded/cookbooks/cache/cookbooks/acme/resources/certificate.rb

        108:     ruby_block "create certificate for #{new_resource.cn}" do # ~FC014
        109:       block do
        110:         unless (all_validations.map { |authz| authz.status == 'valid' }).all?
        111:           errors = all_validations.select do |authz|
        112:             authz.status != 'valid'
        113:           end.map do |authz|
        114:             "{url: #{authz.url}, status: #{authz.status}, error: #{authz.error}} "
        115:           end.reduce(:+)
        116:
        117:           fail "[#{new_resource.cn}] Validation failed, unable to request certificate, Errors: [#{errors}]"
        118:         end
        119:
        120:         begin
        121:           newcert = acme_cert(order, new_resource.cn, mykey, new_resource.alt_names)
        122:         rescue Acme::Client::Error => e
        123:           fail "[#{new_resource.cn}] Certificate request failed: #{e.message}"
        124:         else
        125:           Chef::Resource::File.new("#{new_resource.cn} SSL new crt", run_context).tap do |f|
        126:             f.path    new_resource.crt
        127:             f.owner   new_resource.owner
        128:             f.group   new_resource.group
        129:             f.content newcert
        130:             f.mode    00644
        131:           end.run_action :create
        132:         end
        133:       end
        134:     end
        135:   end
        136: end

        Compiled Resource:
        ------------------
        # Declared in /opt/gitlab/embedded/cookbooks/cache/cookbooks/acme/resources/certificate.rb:108:in `block in class_from_file'

        ruby_block("create certificate for git.rubberbandgames.com") do
          action [:run]
          default_guard_interpreter :default
          declared_type :ruby_block
          cookbook_name "letsencrypt"
          block #<Proc:0x00005615a64c2130 /opt/gitlab/embedded/cookbooks/cache/cookbooks/acme/resources/certificate.rb:109>
          block_name "create certificate for git.rubberbandgames.com"
        end

        System Info:
        ------------
        chef_version=15.17.4
        platform=ubuntu
        platform_version=20.04
        ruby=ruby 2.7.5p203 (2021-11-24 revision f69aeb8314) [x86_64-linux]
        program_name=/opt/gitlab/embedded/bin/chef-client
        executable=/opt/gitlab/embedded/bin/chef-client


      ================================================================================
      Error executing action `create` on resource 'acme_certificate[staging]'
      ================================================================================

      RuntimeError
      ------------
      ruby_block[create certificate for git.rubberbandgames.com] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/acme/resources/certificate.rb line 108) had an error: RuntimeError: [git.rubberbandgames.com] Validation failed, unable to request certificate, Errors: [{url: https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/1226773828/Z2irTg, status: invalid, error: {"type"=>"urn:ietf:params:acme:error:connection", "detail"=>"Fetching http://git.rubberbandgames.com/.well-known/acme-challenge/E8NuYYNIsIvXUPyEfy64L5fwT2cZbCr92m8CXgJyUC0: Timeout during connect (likely firewall problem)", "status"=>400}} ]

      Cookbook Trace:
      ---------------
      /opt/gitlab/embedded/cookbooks/cache/cookbooks/acme/resources/certificate.rb:117:in `block (3 levels) in class_from_file'

      Resource Declaration:
      ---------------------
      suppressed sensitive resource output

      Compiled Resource:
      ------------------
      suppressed sensitive resource output

      System Info:
      ------------
      chef_version=15.17.4
      platform=ubuntu
      platform_version=20.04
      ruby=ruby 2.7.5p203 (2021-11-24 revision f69aeb8314) [x86_64-linux]
      program_name=/opt/gitlab/embedded/bin/chef-client
      executable=/opt/gitlab/embedded/bin/chef-client


    ================================================================================
    Error executing action `create` on resource 'letsencrypt_certificate[git.rubberbandgames.com]'
    ================================================================================

    RuntimeError
    ------------
    acme_certificate[staging] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/letsencrypt/resources/certificate.rb line 41) had an error: RuntimeError: ruby_block[create certificate for git.rubberbandgames.com] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/acme/resources/certificate.rb line 108) had an error: RuntimeError: [git.rubberbandgames.com] Validation failed, unable to request certificate, Errors: [{url: https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/1226773828/Z2irTg, status: invalid, error: {"type"=>"urn:ietf:params:acme:error:connection", "detail"=>"Fetching http://git.rubberbandgames.com/.well-known/acme-challenge/E8NuYYNIsIvXUPyEfy64L5fwT2cZbCr92m8CXgJyUC0: Timeout during connect (likely firewall problem)", "status"=>400}} ]

    Cookbook Trace:
    ---------------
    /opt/gitlab/embedded/cookbooks/cache/cookbooks/acme/resources/certificate.rb:117:in `block (3 levels) in class_from_file'

    Resource Declaration:
    ---------------------
    # In /opt/gitlab/embedded/cookbooks/cache/cookbooks/letsencrypt/recipes/http_authorization.rb

      6: letsencrypt_certificate site do
      7:   crt node['gitlab']['nginx']['ssl_certificate']
      8:   key node['gitlab']['nginx']['ssl_certificate_key']
      9:   notifies :run, "execute[reload nginx]", :immediate
     10:   notifies :run, 'ruby_block[display_le_message]'
     11:   only_if { omnibus_helper.service_up?('nginx') }
     12: end

    Compiled Resource:
    ------------------
    # Declared in /opt/gitlab/embedded/cookbooks/cache/cookbooks/letsencrypt/recipes/http_authorization.rb:6:in `from_file'

    letsencrypt_certificate("git.rubberbandgames.com") do
      action [:create]
      updated true
      updated_by_last_action true
      default_guard_interpreter :default
      declared_type :letsencrypt_certificate
      cookbook_name "letsencrypt"
      recipe_name "http_authorization"
      crt "/etc/gitlab/ssl/git.rubberbandgames.com.crt"
      key "/etc/gitlab/ssl/git.rubberbandgames.com.key"
      alt_names []
      cn "git.rubberbandgames.com"
      only_if { #code block }
    end

    System Info:
    ------------
    chef_version=15.17.4
    platform=ubuntu
    platform_version=20.04
    ruby=ruby 2.7.5p203 (2021-11-24 revision f69aeb8314) [x86_64-linux]
    program_name=/opt/gitlab/embedded/bin/chef-client
    executable=/opt/gitlab/embedded/bin/chef-client


Running handlers:
There was an error running gitlab-ctl reconfigure:

letsencrypt_certificate[git.rubberbandgames.com] (letsencrypt::http_authorization line 6) had an error: RuntimeError: acme_certificate[staging] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/letsencrypt/resources/certificate.rb line 41) had an error: RuntimeError: ruby_block[create certificate for git.rubberbandgames.com] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/acme/resources/certificate.rb line 108) had an error: RuntimeError: [git.rubberbandgames.com] Validation failed, unable to request certificate, Errors: [{url: https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/1226773828/Z2irTg, status: invalid, error: {"type"=>"urn:ietf:params:acme:error:connection", "detail"=>"Fetching http://git.rubberbandgames.com/.well-known/acme-challenge/E8NuYYNIsIvXUPyEfy64L5fwT2cZbCr92m8CXgJyUC0: Timeout during connect (likely firewall problem)", "status"=>400}} ]
3

So I changed my external-url to the local ip of the device, and now it brings up a webpage when using that local ip.

The web page says “Welcome to nginx!” what happened to my gitlabs page?

4

Okay figured out why it was moaning about a firewall, my DNS on my webserver wasn’t pointed to the correct ip. Glad this tool exists https://letsdebug.net/

Okay next error

There was an error running gitlab-ctl reconfigure:

letsencrypt_certificate[git.rubberbandgames.com] (letsencrypt::http_authorization line 6) had an error: RuntimeError: acme_certificate[staging] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/letsencrypt/resources/certificate.rb line 41) had an error: RuntimeError: ruby_block[create certificate for git.rubberbandgames.com] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/acme/resources/certificate.rb line 108) had an error: RuntimeError: [git.rubberbandgames.com] Validation failed, unable to request certificate, Errors: [{url: https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/1227577588/WqpWlQ, status: invalid, error: {"type"=>"urn:ietf:params:acme:error:unauthorized", "detail"=>"Invalid response from http://git.rubberbandgames.com/.well-known/acme-challenge/aOK-6-NFE8FBmIBLKmf27oriRGtux2Rp1FuRuY_S4Ac [176.253.181.49]: \"<html>\\r\\n<head><title>404 Not Found</title></head>\\r\\n<body>\\r\\n<center><h1>404 Not Found</h1></center>\\r\\n<hr><center>nginx/1.18.0 (Ub\"", "status"=>403}} ]
5

So for some reason when I loaded the website it came up with “Welcome to nginx!” I found a guide on how to fix it, which was to remove the default page or something like that. I’ve done that but now I cannot access the website at all.

It’s weird because I even have Nginx as disabled in the config

I now have a “This site can’t be reached” message - This isn’t going great

6

hai,
I’m facing the same error.

Please help