Gitlab CE LetsEncrypt behind Apache Proxy unable to create certificates

Hello,

I am running Gitlab CE 11.11.0 behind an apache proxy with this config

ServerSignature Off

ProxyPreserveHost On
  # Ensure that encoded slashes are not decoded but left in their encoded state.
  # http://doc.gitlab.com/ce/api/projects.html#get-single-project
  AllowEncodedSlashes NoDecode
  <Location />
    # New authorization commands for apache 2.4 and up
    # http://httpd.apache.org/docs/2.4/upgrading.html#access
    Require all granted
    #Allow forwarding to gitlab-workhorse
    ProxyPassReverse http://127.0.0.1:8181
    ProxyPassReverse http://{DOMAIN}/
  </Location>
  # Apache equivalent of nginx try files
  # http://serverfault.com/questions/290784/what-is-apaches-equivalent-of-nginxs-try-files
  # http://stackoverflow.com/questions/10954516/apache2-proxypass-for-rails-app-gitlab
  RewriteEngine on
  #Forward all requests to gitlab-workhorse except existing files like error documents
  RewriteCond %{DOCUMENT_ROOT}/%{REQUEST_FILENAME} !-f [OR]
  RewriteCond %{REQUEST_URI} ^/uploads/.*
  RewriteRule .* http://127.0.0.1:8181%{REQUEST_URI} [P,QSA,NE]
  # needed for downloading attachments
  #DocumentRoot /opt/gitlab/embedded/service/gitlab-rails/public
  #Set up apache error documents, if back end goes down (i.e. 503 error) then a maintenance/deploy page is thrown up.
  ErrorDocument 404 /404.html
  ErrorDocument 422 /422.html
  ErrorDocument 500 /500.html
  ErrorDocument 502 /502.html
  ErrorDocument 503 /503.html

{DOMAIN} = my gitlab domain (gitlab.paskl.de)

and enabled LE in gitlabconf like so:

## Lets Encrypt
letsencrypt['enable'] = true
letsencrypt['auto_renew_hour'] = "12"
letsencrypt['auto_renew_minute'] = "30"
letsencrypt['auto_renew_day_of_month'] = "*/7"
letsencrypt['auto_renew'] = true

However, when recreating config files with gitlab I receive this error:

================================================================================
    Error executing action `create` on resource 'letsencrypt_certificate[gitlab.paskl.de]'
    ================================================================================

    RuntimeError
    ------------
    acme_certificate[staging] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/letsencrypt/resources/certificate.rb line 20) had an error: RuntimeError: [gitlab.paskl.de] Validation failed for domain gitlab.paskl.de

    Cookbook Trace:
    ---------------
    /opt/gitlab/embedded/cookbooks/cache/cookbooks/acme/providers/certificate.rb:93:in `block (2 levels) in class_from_file'
    /opt/gitlab/embedded/cookbooks/cache/cookbooks/acme/providers/certificate.rb:68:in `map'
    /opt/gitlab/embedded/cookbooks/cache/cookbooks/acme/providers/certificate.rb:68:in `block in class_from_file'

    Resource Declaration:
    ---------------------
    # In /opt/gitlab/embedded/cookbooks/cache/cookbooks/letsencrypt/recipes/http_authorization.rb

      3: letsencrypt_certificate site do
      4:   fullchain node['gitlab']['nginx']['ssl_certificate']
      5:   key node['gitlab']['nginx']['ssl_certificate_key']
      6:   notifies :run, "execute[reload nginx]", :immediate
      7:   notifies :run, 'ruby_block[display_le_message]'
      8: end

    Compiled Resource:
    ------------------
    # Declared in /opt/gitlab/embedded/cookbooks/cache/cookbooks/letsencrypt/recipes/http_authorization.rb:3:in `from_file'

    letsencrypt_certificate("gitlab.paskl.de") do
      action [:create]
      updated true
      updated_by_last_action true
      default_guard_interpreter :default
      declared_type :letsencrypt_certificate
      cookbook_name "letsencrypt"
      recipe_name "http_authorization"
      fullchain "/etc/gitlab/ssl/gitlab.paskl.de.crt"
      key "/etc/gitlab/ssl/gitlab.paskl.de.key"
      alt_names []
      cn "gitlab.paskl.de"
    end

    System Info:
    ------------
    chef_version=13.6.4
    platform=ubuntu
    platform_version=16.04
    ruby=ruby 2.5.3p105 (2018-10-18 revision 65156) [x86_64-linux]
    program_name=/opt/gitlab/embedded/bin/chef-client
    executable=/opt/gitlab/embedded/bin/chef-client


Running handlers:
There was an error running gitlab-ctl reconfigure:

letsencrypt_certificate[gitlab.paskl.de] (letsencrypt::http_authorization line 3) had an error: RuntimeError: acme_certificate[staging] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/letsencrypt/resources/certificate.rb line 20) had an error: RuntimeError: [gitlab.paskl.de] Validation failed for domain gitlab.paskl.de

I think its because of the proxy, so that /.well-known is not correctly forwarded to gitlab in the background, but the apache proxy should just pass every request to gitlab if I am not mistaken.

Does anyone know what could be the error?

Hey, could you please check LetsEncrypt certificates fail in domain validation. I was looking if anybody else had a similiar issue but this is the only thing I could find.

Hello, I’ve tried the suggested solutions from the other thread but it didnt do it for me.

This is my config file (I redacted a few things): https://pastebin.com/hpva5vqY

This is added at the end of the gitlab.rb file

letsencrypt['enable'] = true
letsencrypt['auto_renew'] = true
letsencrypt['auto_renew_hour'] = 0
letsencrypt['auto_renew_minute'] = 30
letsencrypt['auto_renew_day_of_month'] = "*/4"
nginx['custom_gitlab_server_config'] = "location /.well-known/acme-challenge/ {\n root /var/opt/gitlab/nginx/www/; \n}\n"
nginx['redirect_http_to_https'] = true
nginx['redirect_http_to_https_port'] = 80

Note that I disabled (for whatever reason, I cannot remember) “nginx” in the gitlab.rb file - might be an issue with my setup? Because its now Apache (Proxy Redirect) > Gitlab and not Apache (Proxy) > Nginx > Gitlab.