Omnibus installer fails on letsencrypt

I’m installing the omnibus package on Debian 9.4, and it’s failing on a letsencrypt-related error. Unfortunately, it’s not giving me much to go on to troubleshoot the problem.

There’s a lot of output to go through, but this is the only error I found:

Recipe: nginx::enable
  * service[nginx] action restart
    - restart service service[nginx]
Recipe: letsencrypt::http_authorization
  * letsencrypt_certificate[gitlab.redacted] action create
    * acme_certificate[staging] action create
      * file[gitlab.redacted SSL key] action create_if_missing
        - create new file /etc/gitlab/ssl/gitlab.redacted.key-staging
        - update content in file /etc/gitlab/ssl/gitlab.redacted.key-staging from none to f643d2
        - suppressed sensitive resource
        - change mode from '' to '0400'
        - change owner from '' to 'root'
        - change group from '' to 'root'
      * directory[/var/opt/gitlab/nginx/www/.well-known/acme-challenge] action create
        - create new directory /var/opt/gitlab/nginx/www/.well-known/acme-challenge
        - change mode from '' to '0755'
        - change owner from '' to 'root'
        - change group from '' to 'root'
      * file[/var/opt/gitlab/nginx/www/.well-known/acme-challenge/EaamDP28bpVRf6vAM_vAoCqoM3e-39LGaXht1rVi-Wg] action create
        - create new file /var/opt/gitlab/nginx/www/.well-known/acme-challenge/EaamDP28bpVRf6vAM_vAoCqoM3e-39LGaXht1rVi-Wg
        - update content in file /var/opt/gitlab/nginx/www/.well-known/acme-challenge/EaamDP28bpVRf6vAM_vAoCqoM3e-39LGaXht1rVi-Wg from none to 739006
        --- /var/opt/gitlab/nginx/www/.well-known/acme-challenge/EaamDP28bpVRf6vAM_vAoCqoM3e-39LGaXht1rVi-Wg	2018-07-09 23:35:39.836186529 -0400
        +++ /var/opt/gitlab/nginx/www/.well-known/acme-challenge/.chef-EaamDP28bpVRf6vAM_vAoCqoM3e-39LGaXht1rVi-Wg20180709-3655-qtmplm	2018-07-09 23:35:39.836186529 -0400
        @@ -1 +1,2 @@
        +EaamDP28bpVRf6vAM_vAoCqoM3e-39LGaXht1rVi-Wg.-eyHg1xUBaKPChytn95by8jROZVLOWRJ049WhgrPlWg
        - change mode from '' to '0644'
        - change owner from '' to 'root'
        - change group from '' to 'root'

      ================================================================================
      Error executing action `create` on resource 'acme_certificate[staging]'
      ================================================================================

      RuntimeError
      ------------
      [gitlab.redacted] Validation failed for domain gitlab.redacted

      Cookbook Trace:
      ---------------
      /opt/gitlab/embedded/cookbooks/cache/cookbooks/acme/providers/certificate.rb:93:in `block (2 levels) in class_from_file'
      /opt/gitlab/embedded/cookbooks/cache/cookbooks/acme/providers/certificate.rb:68:in `map'
      /opt/gitlab/embedded/cookbooks/cache/cookbooks/acme/providers/certificate.rb:68:in `block in class_from_file'

      Resource Declaration:
      ---------------------
      suppressed sensitive resource output

      Compiled Resource:
      ------------------
      suppressed sensitive resource output

      System Info:
      ------------
      chef_version=13.6.4
      platform=debian
      platform_version=9.4
      ruby=ruby 2.4.4p296 (2018-03-28 revision 63013) [x86_64-linux]
      program_name=/opt/gitlab/embedded/bin/chef-client
      executable=/opt/gitlab/embedded/bin/chef-client


    ================================================================================
    Error executing action `create` on resource 'letsencrypt_certificate[gitlab.redacted]'
    ================================================================================

    RuntimeError
    ------------
    acme_certificate[staging] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/letsencrypt/resources/certificate.rb line 20) had an error: RuntimeError: [gitlab.redacted] Validation failed for domain gitlab.redacted

    Cookbook Trace:
    ---------------
    /opt/gitlab/embedded/cookbooks/cache/cookbooks/acme/providers/certificate.rb:93:in `block (2 levels) in class_from_file'
    /opt/gitlab/embedded/cookbooks/cache/cookbooks/acme/providers/certificate.rb:68:in `map'
    /opt/gitlab/embedded/cookbooks/cache/cookbooks/acme/providers/certificate.rb:68:in `block in class_from_file'

    Resource Declaration:
    ---------------------
    # In /opt/gitlab/embedded/cookbooks/cache/cookbooks/letsencrypt/recipes/http_authorization.rb

      3: letsencrypt_certificate site do
      4:   fullchain node['gitlab']['nginx']['ssl_certificate']
      5:   key node['gitlab']['nginx']['ssl_certificate_key']
      6:   notifies :run, "execute[reload nginx]", :immediate
      7:   notifies :run, 'ruby_block[display_le_message]'
      8: end

    Compiled Resource:
    ------------------
    # Declared in /opt/gitlab/embedded/cookbooks/cache/cookbooks/letsencrypt/recipes/http_authorization.rb:3:in `from_file'

    letsencrypt_certificate("gitlab.redacted") do
      action [:create]
      updated true
      updated_by_last_action true
      default_guard_interpreter :default
      declared_type :letsencrypt_certificate
      cookbook_name "letsencrypt"
      recipe_name "http_authorization"
      fullchain "/etc/gitlab/ssl/gitlab.redacted.crt"
      key "/etc/gitlab/ssl/gitlab.redacted.key"
      alt_names []
      cn "gitlab.redacted"
    end

    System Info:
    ------------
    chef_version=13.6.4
    platform=debian
    platform_version=9.4
    ruby=ruby 2.4.4p296 (2018-03-28 revision 63013) [x86_64-linux]
    program_name=/opt/gitlab/embedded/bin/chef-client
    executable=/opt/gitlab/embedded/bin/chef-client

And then the deb install bombs out at the end:

Recipe: gitlab::postgres-exporter
  * service[postgres-exporter] action restart
    - restart service service[postgres-exporter]
  * ruby_block[restart postgres-exporter svlogd configuration] action create
    - execute the ruby block restart postgres-exporter svlogd configuration
  * ruby_block[reload postgres-exporter svlogd configuration] action create
    - execute the ruby block reload postgres-exporter svlogd configuration

Running handlers:
There was an error running gitlab-ctl reconfigure:

letsencrypt_certificate[gitlab.redacted] (letsencrypt::http_authorization line 3) had an error: RuntimeError: acme_certificate[staging] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/letsencrypt/resources/certificate.rb line 20) had an error: RuntimeError: [gitlab.redacted] Validation failed for domain gitlab.redacted

Running handlers complete
Chef Client failed. 433 resources updated in 02 minutes 54 seconds
dpkg: error processing package gitlab-ee (--configure):
 subprocess installed post-installation script returned error exit status 1
Errors were encountered while processing:
 gitlab-ee
E: Sub-process /usr/bin/dpkg returned an error code (1)

So it’s failing the letsencrypt validation, but I don’t know why. I can’t find any logs or anything with more detail on the error.

I can probably work around it by installing without ssl and then converting the site over, but a secure install seemed like a thing that I should expect to work.

Any suggestions for where to look for issues?

I had a similar problem but unfortunately it has been too long for me to recall.

A few suggestions to try:

  1. Since it’s a validation error, make sure you have ports 80 and 443 open, and make sure your domain is pointing at your server and the dns change has been given time to propagate.
  2. If that doesn’t work, someone else oddly enough had success running sudo service docker stop (if you happened to have it installed. I have no idea about why this one would work…)
  3. My final suggestion would be to check if port 80 is in use by something else (running something like netstat -an | grep “:80” should let you know).

Sorry. They aren’t great suggestions, but I didn’t spot any immediate tells in the debug code. Good luck!

I realized I never circled back on this.

I ended up configuring the site as HTTP instead of HTTPS, and then going back and converting it after the installer was done. I was planning to use a wildcard certificate we have anyway, rather than the LetsEncrypt cert, so it wasn’t a major problem.

It seems unhelpful that the installer doesn’t pass on an error message indicating what the problem is, though.