Let's Encrypt certificate with Pages hosted on a different domain?

I have a self-hosted GitLab site running at https://git.hatchbed.com, using Let’s Encrypt to automatically obtain a certificate. It’s been working fine. I would like to set up GitLab to host Pages under a different domain, hatchbed.io (the registrar for hatchbed.com doesn’t allow wildcards in records and we don’t want to transfer the domain), but when I try to change our gitlab.rb configuration to expose pages over https://hatchbed.io , nginx fails to start due a missing certificate, and it doesn’t seem to make any attempt to use Let’s Encrypt to obtain one. If I configure it to use http://hatchbed.io instead, that works, but obviously that isn’t desirable. Is there any way to make GitLab use Let’s Encrypt to obtain a certificate for that domain, or am I going to have to manually manage it?

For reference, the base site in /etc/gitlab/gitlab.rb is defined like this:

external_url 'https://git.hatchbed.com'

And my Pages-related configuration in /etc/gitlab/gitlab.rb is very simple, it just looks like this:

##! Define to enable GitLab Pages
pages_external_url "https://hatchbed.io"
gitlab_pages['enable'] = true

Running gitlab-ctl reconfigure fails in the runit_service[nginx] step; here’s the error it prints

Recipe: nginx::enable                                                                                                                                                       * runit_service[nginx] action restart                                                                                                                                                                                                                                                                                                                 ================================================================================                                                                                          Error executing action `restart` on resource 'runit_service[nginx]'                                                                                                       ================================================================================                                                                                                                                                                                                                                                                    Mixlib::ShellOut::ShellCommandFailed                                                                                                                                      ------------------------------------                                                                                                                                      Expected process to exit with [0], but received '1'                                                                                                                       ---- Begin output of /opt/gitlab/embedded/bin/sv restart /opt/gitlab/service/nginx ----                                                                                   STDOUT: timeout: down: /opt/gitlab/service/nginx: 1s, normally up, want up                                                                                                STDERR:                                                                                                                                                               
    ---- End output of /opt/gitlab/embedded/bin/sv restart /opt/gitlab/service/nginx ----                                                                                 
    Ran /opt/gitlab/embedded/bin/sv restart /opt/gitlab/service/nginx returned 1                                                                                          
                                                                                                                                                                          
    Cookbook Trace:                                                                                                                                                       
    ---------------                                                                                                                                                       
    /opt/gitlab/embedded/cookbooks/cache/cookbooks/runit/libraries/helpers.rb:136:in `tap'                                                                                
    /opt/gitlab/embedded/cookbooks/cache/cookbooks/runit/libraries/helpers.rb:136:in `safe_sv_shellout!'                                                                  
    /opt/gitlab/embedded/cookbooks/cache/cookbooks/runit/libraries/helpers.rb:164:in `restart_service'                                                                    
    /opt/gitlab/embedded/cookbooks/cache/cookbooks/runit/libraries/provider_runit_service.rb:358:in `block in <class:RunitService>'                                       
                                                                                                                                                                          
    Resource Declaration:                                                                                                                                                 
    ---------------------                                                                                                                                                 
    # In /opt/gitlab/embedded/cookbooks/cache/cookbooks/nginx/recipes/enable.rb                                                                                           
                                                                                                                                                                          
     18: runit_service "nginx" do                                                                                                                                         
     19:   start_down node['gitlab']['nginx']['ha']                                                                                                                       
     20:   options({                                                                                                                                                      
     21:     log_directory: nginx_log_dir                                                                                                                                 
     22:   }.merge(params))                                                                                                                                               
     23:   log_options node['gitlab']['logging'].to_hash.merge(node['gitlab']['nginx'].to_hash)                                                                           
     24: end                                                                                                                                                              
     25:                                                                                                                                                                  
                                                                                                                                                                          
    Compiled Resource:                                                                                                                                                    
    ------------------                                                                                                                                                    
    # Declared in /opt/gitlab/embedded/cookbooks/cache/cookbooks/nginx/recipes/enable.rb:18:in `from_file'

    runit_service("nginx") do                                                                                                                                                   action [:enable]                                                                                                                                                          default_guard_interpreter :default                                                                                                                                        declared_type :runit_service                                                                                                                                              cookbook_name "nginx"                                                                                                                                                     recipe_name "enable"                                                                                                                                                      start_down false                                                                                                                                                          service_name "nginx"                                                                                                                                                      options {:env_dir=>"/opt/gitlab/sv/nginx/env", :log_directory=>"/var/log/gitlab/nginx"}                                                                                   log_options {"svlogd_size"=>209715200, "svlogd_num"=>30, "svlogd_timeout"=>86400, "svlogd_filter"=>"gzip", "svlogd_udp"=>nil, "svlogd_prefix"=>nil, "udp_log_shipping_host"=>nil, "udp_log_shipping_hostname"=>nil, "udp_log_shipping_port"=>514, "logrotate_frequency"=>"daily", "logrotate_maxsize"=>nil, "logrotate_size"=>nil, "logrotate_rotate"=>30, "logrotate_compress"=>"compress", "logrotate_method"=>"copytruncate", "logrotate_postrotate"=>nil, "logrotate_dateformat"=>nil, "enable"=>true, "ha"=>false, "dir"=>"/var/opt/gitlab/nginx", "log_directory"=>"/var/log/gitlab/nginx", "error_log_level"=>"error", "worker_processes"=>4, "worker_connections"=>10240, "log_format"=>"$remote_addr - $remote_user [$time_local] \"$request_method $filtered_request_uri $server_protocol\" $status $body_bytes_sent \"$filtered_http_referer\" \"$http_user_agent\" $gzip_ratio", "sendfile"=>"on", "tcp_nopush"=>"on", "tcp_nodelay"=>"on", "hide_server_tokens"=>"off", "gzip_http_version"=>"1.1", "gzip_comp_level"=>"2", "gzip_proxied"=>"no-cache no-store private expired auth", "gzip_types"=>["text/plain", "text/css", "application/x-javascript", "text/xml", "application/xml", "application/xml+rss", "t
ext/javascript", "application/json"], "keepalive_timeout"=>65, "keepalive_time"=>"1h", "client_max_body_size"=>0, "cache_max_size"=>"5000m", "redirect_http_to_https"=>fal
se, "redirect_http_to_https_port"=>80, "request_buffering_off_path_regex"=>"/api/v\\d/jobs/\\d+/artifacts$|\\.git/git-receive-pack$|\\.git/gitlab-lfs/objects|\\.git/info/
lfs/objects/batch$", "ssl_client_certificate"=>nil, "ssl_verify_client"=>nil, "ssl_verify_depth"=>"1", "ssl_certificate"=>"/etc/gitlab/ssl/git.hatchbed.com.crt", "ssl_cer
tificate_key"=>"/etc/gitlab/ssl/git.hatchbed.com.key", "ssl_ciphers"=>"ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-A
ES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:AES256-GCM-SHA384", "ssl_prefer_server_ciph
ers"=>"off", "ssl_protocols"=>"TLSv1.2 TLSv1.3", "ssl_session_cache"=>"shared:SSL:10m", "ssl_session_tickets"=>"off", "ssl_session_timeout"=>"1d", "ssl_dhparam"=>nil, "li
sten_addresses"=>["*"], "listen_port"=>443, "listen_https"=>nil, "custom_gitlab_server_config"=>nil, "custom_nginx_config"=>nil, "proxy_read_timeout"=>3600, "proxy_connec
t_timeout"=>300, "proxy_set_headers"=>{"Host"=>"$http_host_with_default", "X-Real-IP"=>"$remote_addr", "X-Forwarded-For"=>"$proxy_add_x_forwarded_for", "Upgrade"=>"$http_
upgrade", "Connection"=>"$connection_upgrade", "X-Forwarded-Proto"=>"https", "X-Forwarded-Ssl"=>"on"}, "proxy_protocol"=>false, "proxy_custom_buffer_size"=>nil, "referrer
_policy"=>"strict-origin-when-cross-origin", "http2_enabled"=>true, "proxy_cache_path"=>"proxy_cache keys_zone=gitlab:10m max_size=1g levels=1:2", "proxy_cache"=>"gitlab"
, "real_ip_trusted_addresses"=>[], "real_ip_header"=>nil, "real_ip_recursive"=>nil, "server_names_hash_bucket_size"=>64, "hsts_max_age"=>63072000, "hsts_include_subdomain
s"=>false, "gzip_enabled"=>true, "consul_service_name"=>"nginx", "consul_service_meta"=>nil, "status"=>{"enable"=>true, "listen_addresses"=>["*"], "fqdn"=>"localhost", "p
ort"=>8060, "vts_enable"=>true, "options"=>{"server_tokens"=>"off", "access_log"=>"off", "allow"=>"127.0.0.1", "deny"=>"all"}}}                                           
      run_template_name "nginx"                                                                                                                                           
      log_template_name "nginx"                                                                                                                                           
      check_script_template_name "nginx"                                                                                                                                  
      finish_script_template_name "nginx"                                                                                                                                 
      control_template_names {}                                                                                                                                           
    end                                                                                                                                                                   
                                                                                                                                                                          
    System Info:                                                                                                                                                          
    ------------                                                                                                                                                          
    chef_version=15.17.4                                                                                                                                                  
    platform=ubuntu                                                                                                                                                       
    platform_version=20.04                                                                                                                                                
    ruby=ruby 2.7.5p203 (2021-11-24 revision f69aeb8314) [x86_64-linux]                                                                                                   
    program_name=/opt/gitlab/embedded/bin/chef-client                                                                                                                     
    executable=/opt/gitlab/embedded/bin/chef-client                                                                                                                       
                                                                                                                                                                          
Recipe: letsencrypt::enable                                                                                                                                               
  * ruby_block[display_le_message] action run                                                                                                                             
    - execute the ruby block display_le_message

Running handlers:                                                                                                                                                         
There was an error running gitlab-ctl reconfigure:                                                                                                                        
                                                                                                                                                                          
runit_service[nginx] (nginx::enable line 18) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'                      
---- Begin output of /opt/gitlab/embedded/bin/sv restart /opt/gitlab/service/nginx ----                                                                                   
STDOUT: timeout: down: /opt/gitlab/service/nginx: 1s, normally up, want up                                                                                                
STDERR:                                                                                                                                                                   
---- End output of /opt/gitlab/embedded/bin/sv restart /opt/gitlab/service/nginx ----                                                                                     
Ran /opt/gitlab/embedded/bin/sv restart /opt/gitlab/service/nginx returned 1                                                                                              
                                                                                                                                                                          
Running handlers complete                                                                                                                                                 
Chef Infra Client failed. 15 resources updated in 01 minutes 35 seconds 

After that, nginx doesn’t respond, and the log file at /var/log/gitlab/nginx/error.log is filled with errors like this:

2022/05/19 15:58:34 [emerg] 3140174#0: cannot load certificate "/etc/gitlab/ssl/hatchbed.io.crt": BIO_new_file() failed (SSL: error:02001002:system library:fopen:No such file or directory:fopen('/etc/gitlab/ssl/hatchbed.io.crt','r') error:2006D080:BIO routines:BIO_new_file:no such file)

There is indeed no certificate at /etc/gitlab/ssl/hatchbed.io.crt. The log file under /var/log/gitlab/lets-encrypt was last updated when it last update the certificate for git.hatchbed.com; I don’t see any indication that it’s even tried to retrieve one for hatchbed.io.

1 Like