My 1+ year perfectly working Let’s Encrypt integration stopped working some weeks ago.
The error message is:
RuntimeError: letsencrypt_certificate[git.example.com] (letsencrypt::http_authorization line 5) had an error: RuntimeError: acme_certificate[staging] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/letsencrypt/resources/certificate.rb line 25) had an error: RuntimeError: ruby_block[create certificate for git.example.com] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/acme/resources/certificate.rb line 108) had an error: RuntimeError: [git.example.com] Validation failed, unable to request certificate
The log files read:
# Logfile created on 2019-12-22 15:14:24 +0000 by logger.rb/66358
[2019-12-22T15:14:24+00:00] INFO: Started chef-zero at chefzero://localhost:1 with repository at /opt/gitlab/embedded One version per cookbook
[2019-12-22T15:14:24+00:00] INFO: *** Chef 14.13.11 ***
[2019-12-22T15:14:24+00:00] INFO: Platform: x86_64-linux
[2019-12-22T15:14:24+00:00] INFO: Chef-client pid: 2415
[2019-12-22T15:14:24+00:00] INFO: The plugin path /etc/chef/ohai/plugins does not exist. Skipping...
[2019-12-22T15:14:25+00:00] INFO: Setting the run_list to ["recipe[gitlab::letsencrypt_renew]"] from CLI options
[2019-12-22T15:14:25+00:00] INFO: Run List is [recipe[gitlab::letsencrypt_renew]]
[2019-12-22T15:14:25+00:00] INFO: Run List expands to [gitlab::letsencrypt_renew]
[2019-12-22T15:14:25+00:00] INFO: Starting Chef Run for gitlab-zeta
[2019-12-22T15:14:25+00:00] INFO: Running start handlers
[2019-12-22T15:14:25+00:00] INFO: Start handlers complete.
[2019-12-22T15:14:26+00:00] INFO: Loading cookbooks [gitlab@0.0.1, package@0.1.0, postgresql@0.1.0, redis@0.1.0, monitoring@0.1.0, registry@0.1.0, mattermost@0.1.0, consul@0.1.0, gitaly@0.1.0, praefect@0.1.0, letsencrypt@0.1.0, nginx@0.1.0, runit@4.3.0,acme@4.0.0, crond@0.1.0]
[2019-12-22T15:14:30+00:00] INFO: file[/var/opt/gitlab/nginx/www/.well-known/acme-challenge/i1ehqtnjiikzGJn_WZixffEHJ0EnFF8ou6BIJHdoE40] created file /var/opt/gitlab/nginx/www/.well-known/acme-challenge/i1ehqtnjiikzGJn_WZixffEHJ0EnFF8ou6BIJHdoE40
[2019-12-22T15:14:30+00:00] INFO: file[/var/opt/gitlab/nginx/www/.well-known/acme-challenge/i1ehqtnjiikzGJn_WZixffEHJ0EnFF8ou6BIJHdoE40] updated file contents /var/opt/gitlab/nginx/www/.well-known/acme-challenge/i1ehqtnjiikzGJn_WZixffEHJ0EnFF8ou6BIJHdoE40
[2019-12-22T15:14:30+00:00] INFO: file[/var/opt/gitlab/nginx/www/.well-known/acme-challenge/i1ehqtnjiikzGJn_WZixffEHJ0EnFF8ou6BIJHdoE40] owner changed to 0
[2019-12-22T15:14:30+00:00] INFO: file[/var/opt/gitlab/nginx/www/.well-known/acme-challenge/i1ehqtnjiikzGJn_WZixffEHJ0EnFF8ou6BIJHdoE40] group changed to 0
[2019-12-22T15:14:30+00:00] INFO: file[/var/opt/gitlab/nginx/www/.well-known/acme-challenge/i1ehqtnjiikzGJn_WZixffEHJ0EnFF8ou6BIJHdoE40] mode changed to 644
[2019-12-22T15:14:33+00:00] INFO: file[/var/opt/gitlab/nginx/www/.well-known/acme-challenge/i1ehqtnjiikzGJn_WZixffEHJ0EnFF8ou6BIJHdoE40] deleted file at /var/opt/gitlab/nginx/www/.well-known/acme-challenge/i1ehqtnjiikzGJn_WZixffEHJ0EnFF8ou6BIJHdoE40
[2019-12-22T15:14:33+00:00] INFO: Running queued delayed notifications before re-raising exception
[2019-12-22T15:14:33+00:00] INFO: Running queued delayed notifications before re-raising exception
[2019-12-22T15:14:33+00:00] INFO: Running queued delayed notifications before re-raising exception
[2019-12-22T15:14:33+00:00] ERROR: Running exception handlers
[2019-12-22T15:14:33+00:00] ERROR: Exception handlers complete
[2019-12-22T15:14:33+00:00] FATAL: Stacktrace dumped to /opt/gitlab/embedded/cookbooks/cache/chef-stacktrace.out
[2019-12-22T15:14:33+00:00] FATAL: Please provide the contents of the stacktrace.out file if you file a bug report
[2019-12-22T15:14:33+00:00] FATAL: RuntimeError: letsencrypt_certificate[git.example.com] (letsencrypt::http_authorization line 5) had an error: RuntimeError: acme_certificate[staging] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/letsencrypt/resources/certificate.rb line 25) had an error: RuntimeError: ruby_block[create certificate for git.example.com] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/acme/resources/certificate.rb line 108) had an error: RuntimeError: [git.example.com] Validation failed, unable to request certificate
I’ve checked whether my site is public available (it is), and whether any CAA record in the DNS prohibits Let’s Encrypt (it doesn’t).
My 1+ year unchanged Let’s Encrypt configuration in /etc/gitlab/gitlab.rb reads:
letsencrypt['enable'] = true
letsencrypt['contact_emails'] = ['user@example.com']
letsencrypt['auto_renew'] = true
My question
Any idea on how to investigate further to fix this error?