Letsencrypt acme-challenge not created for additional gitlab-pages domain

How to Use GitLab Self-managed
1

Hi there,

I’m setting up gitlab pages to be hosted on the same machine as gitlab itself and without enabled domain, it all runs as expected. But, after verifying the domain, I cannot get the letsencrypt certificate.

tl;dr

  • setup pages → ok
  • setup extra domain
    • setup → ok (found no errors via gitlab-ctl tail)
    • verify → ok
    • letsencrypt → fails, challenge file not found

Setup - pages do work, but no extra domain

(1) gitlab.rb

All active lines in gitlab.rb.

# main domain
# team.example.com
external_url 'https://team.example.com'
pages_external_url "https://team.example.com/"
gitlab_pages['enable'] = true
gitlab_pages['metrics_address'] = ":9235"
gitlab_pages['access_control'] = true
mattermost_external_url 'https://talk.example.com'
mattermost['team_site_name'] = "company Mattermost"
alertmanager['flags'] = {
  'web.listen-address' => "localhost:9093",
  'storage.path' => "/var/opt/gitlab/alertmanager/data",
  'config.file' => "/var/opt/gitlab/alertmanager/alertmanager.yml",
  'cluster.advertise-address' => "127.0.0.1:9093"
}
letsencrypt['enable'] = true
letsencrypt['contact_emails'] = ['my.name@example.com','mail@other.com'] # This should be an array of email addresses to add as contacts

(2) Project Config

The project main URL is team.example.com/documentation/it, it’s Pages URL is https://documentation.pages.example.com/it

(3) DNS config

The following DNS records are defined for the site

# DNS records
# A.B.C.D main IP4 address
# E.F.G.H secondary IP4 address
# W:X:Y:Z::1 main IPv6 address

*.team 2400 IN CNAME team
_gitlab-pages-verification-code.documentation 2400 IN TXT "gitlab-pages-verification-code=********************"
dokumentation 2400 IN E.F.G.H
team 2400 IN A A.B.C.D
team 2400 IN AAAA W:X:Y:Z::1

(4) Pages Config

  • HTTPS enforced

Effect

  • Pages opens and displays at https://documentation.team.example.com/it
  • As expected, the certificate is flagged (it’s for team.example.com, not the sub-subdomain)

Failed Domain setup

Updated gitlab.rb to allow Pages domains

# main domain
# team.example.com

external_url 'https://team.example.com'
pages_external_url "https://team.example.com/"
gitlab_pages['enable'] = true

# additional two (2) lines added to gitlab.rb
# point external http to secondary ip address
gitlab_pages['external_http'] = ['E.F.G.H:80']
gitlab_pages['external_https'] = ['E.F.G.H:443']

gitlab_pages['metrics_address'] = ":9235"
gitlab_pages['access_control'] = true
mattermost_external_url 'https://reden.example.com'
mattermost['team_site_name'] = "company Mattermost"
alertmanager['flags'] = {
  'web.listen-address' => "localhost:9093",
  'storage.path' => "/var/opt/gitlab/alertmanager/data",
  'config.file' => "/var/opt/gitlab/alertmanager/alertmanager.yml",
  'cluster.advertise-address' => "127.0.0.1:9093"
}
letsencrypt['enable'] = true
letsencrypt['contact_emails'] = ['my.name@example.com','mail@other.com'] # This should be an array of email addresses to add as contacts

Effect

After reconfigure, the domain can be added and verified:

image
image818×920 70.2 KB

Observations

  • opening documentation.example.com simply opens gitlab, but not Pages content
  • /var/log/gitlab/nginx/gitlab_pages_error.log shows errors during the challenge

error messages

# looking for a specific request
$ grep yYv2B8iGa4HhHLBlh0prWEtkkiBJ2mPYnZ9-qFbZaQM /var/log/gitlab/*/*

/var/log/gitlab/gitlab-rails/application_json.log:{"severity":"ERROR","time":"2022-07-04T06:39:25.660Z","correlation_id":"1381e3aeccd69f2b34ecda81b4657d88","message":"Failed to obtain Let's Encrypt certificate","acme_error":{"type":"urn:ietf:params:acme:error:unauthorized","detail":"116.202.185.178: Invalid response from http://documentation.example.com/.well-known/acme-challenge/yYv2B8iGa4HhHLBlh0prWEtkkiBJ2mPYnZ9-qFbZaQM: 404","status":403},"project_id":2,"pages_domain":"documentation.example.com"}
/var/log/gitlab/gitlab-rails/application.log:2022-07-04T06:39:25.660Z: {:message=>"Failed to obtain Let's Encrypt certificate", :acme_error=>{"type"=>"urn:ietf:params:acme:error:unauthorized", "detail"=>"116.202.185.178: Invalid response from http://documentation.example.com/.well-known/acme-challenge/yYv2B8iGa4HhHLBlh0prWEtkkiBJ2mPYnZ9-qFbZaQM: 404", "status"=>403}, :project_id=>2, :pages_domain=>"documentation.example.com"}
/var/log/gitlab/nginx/gitlab_access.log:A1.B.C.D - - [04/Jul/2022:08:38:22 +0200] "GET /.well-known/acme-challenge/yYv2B8iGa4HhHLBlh0prWEtkkiBJ2mPYnZ9-qFbZaQM HTTP/1.1" 404 117 "" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)" 1.38
/var/log/gitlab/nginx/gitlab_access.log:A2.B.C.D - - [04/Jul/2022:08:38:23 +0200] "GET /.well-known/acme-challenge/yYv2B8iGa4HhHLBlh0prWEtkkiBJ2mPYnZ9-qFbZaQM HTTP/1.1" 404 117 "" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)" 1.38
/var/log/gitlab/nginx/gitlab_access.log:A3.B.C.D - - [04/Jul/2022:08:38:23 +0200] "GET /.well-known/acme-challenge/yYv2B8iGa4HhHLBlh0prWEtkkiBJ2mPYnZ9-qFbZaQM HTTP/1.1" 404 117 "" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)" 1.38
/var/log/gitlab/nginx/gitlab_access.log:A3.B.C.D - - [04/Jul/2022:08:38:23 +0200] "GET /.well-known/acme-challenge/yYv2B8iGa4HhHLBlh0prWEtkkiBJ2mPYnZ9-qFbZaQM HTTP/1.1" 404 117 "" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)" 1.38
/var/log/gitlab/nginx/gitlab_error.log:2022/07/04 08:38:22 [error] 464163#0: *52132 open() "/var/opt/gitlab/nginx/www/.well-known/acme-challenge/yYv2B8iGa4HhHLBlh0prWEtkkiBJ2mPYnZ9-qFbZaQM" failed (2: No such file or directory), client: A1.B.C.D, server: team.example.com, request: "GET /.well-known/acme-challenge/yYv2B8iGa4HhHLBlh0prWEtkkiBJ2mPYnZ9-qFbZaQM HTTP/1.1", host: "documentation.example.com"
/var/log/gitlab/nginx/gitlab_error.log:2022/07/04 08:38:23 [error] 464163#0: *52133 open() "/var/opt/gitlab/nginx/www/.well-known/acme-challenge/yYv2B8iGa4HhHLBlh0prWEtkkiBJ2mPYnZ9-qFbZaQM" failed (2: No such file or directory), client: A2.B.C.D, server: team.example.com, request: "GET /.well-known/acme-challenge/yYv2B8iGa4HhHLBlh0prWEtkkiBJ2mPYnZ9-qFbZaQM HTTP/1.1", host: "documentation.example.com"
/var/log/gitlab/nginx/gitlab_error.log:2022/07/04 08:38:23 [error] 464163#0: *52134 open() "/var/opt/gitlab/nginx/www/.well-known/acme-challenge/yYv2B8iGa4HhHLBlh0prWEtkkiBJ2mPYnZ9-qFbZaQM" failed (2: No such file or directory), client: A3.B.C.D, server: team.example.com, request: "GET /.well-known/acme-challenge/yYv2B8iGa4HhHLBlh0prWEtkkiBJ2mPYnZ9-qFbZaQM HTTP/1.1", host: "documentation.example.com"
/var/log/gitlab/nginx/gitlab_error.log:2022/07/04 08:38:23 [error] 464163#0: *52135 open() "/var/opt/gitlab/nginx/www/.well-known/acme-challenge/yYv2B8iGa4HhHLBlh0prWEtkkiBJ2mPYnZ9-qFbZaQM" failed (2: No such file or directory), client: A3.B.C.D, server: team.example.com, request: "GET /.well-known/acme-challenge/yYv2B8iGa4HhHLBlh0prWEtkkiBJ2mPYnZ9-qFbZaQM HTTP/1.1", host: "documentation.example.com"

Asessment

So far, this looks like the challenge file isn’t created at all, while everything else seems to be working.

Any advise will be greatly appreciated!

Thanks,
Volker