I have a gitlab instance I self-host. For about 6 months now one repository that has pages setup refuses to successfully fetch a LE cert. The odd thing is I have dozens of other pages hosted and they fetch LE just fine. In fact I had cloned the broken repo and created a new domain name to test it out and the cloned repo had no trouble fetching the cert at the new domain name.
The broken domain name is https://jeffreyfreeman.me
The working clone is at https://beta.jeffreyfreeman.me
At one point the cert WAS working on the main site but once it expired it errored out anytime it tried to renew since.
Here is the repo itself for the broken domain: Sign in · GitLab
Finally when I do a “retry” from the settings/pages window to fetch the cert again it errors and produces the following in the logs.
Just for clarity I am not blacklisted on LE and here is the support thread from LE where I got some help but ultimately they were stumped: Is my domain blacklisted? - Help - Let's Encrypt Community Support
2021-07-08T15:44:17.963931146Z ==> /var/log/gitlab/gitlab-rails/production.log <==
2021-07-08T15:44:17.963935706Z Started GET "/-/acme-challenge?domain=jeffreyfreeman.me&token=[FILTERED]" for 17.58.92.204 at 2021-07-08 15:44:17 +0000
2021-07-08T15:44:17.963940576Z Processing by AcmeChallengesController#show as HTML
2021-07-08T15:44:17.963944936Z Parameters: {"domain"=>"jeffreyfreeman.me", "token"=>"[FILTERED]"}
2021-07-08T15:44:17.963950176Z Completed 404 Not Found in 8ms (ActiveRecord: 2.4ms | Elasticsearch: 0.0ms | Allocations: 470)
2021-07-08T15:44:17.963954496Z
2021-07-08T15:44:17.963958547Z ==> /var/log/gitlab/gitlab-workhorse/current <==
2021-07-08T15:44:17.963963337Z {"content_type":"text/html; charset=utf-8","correlation_id":"CFKuNzhCjB9","duration_ms":43,"host":"git.qoto.org","level":"info","method":"GET","msg":"access","proto":"HTTP/1.1","referrer":"","remote_addr":"17.58.92.204:0","remote_ip":"17.58.92.204","status":404,"system":"http","time":"2021-07-08T15:44:17Z","uri":"/-/acme-challenge?domain=jeffreyfreeman.me\u0026token=[FILTERED]","user_agent":"AppleNewsBot","written_bytes":3108}
2021-07-08T15:44:17.963972947Z
2021-07-08T15:44:17.963977117Z ==> /var/log/gitlab/nginx/gitlab_access.log <==
2021-07-08T15:44:17.963981917Z 192.168.0.6 - - [08/Jul/2021:15:44:17 +0000] "GET /-/acme-challenge?domain=jeffreyfreeman.me&token=pKAsjNmT6hfhhR1-Kv0xcWhFv_y9GseqxxRv4sgAi1M HTTP/1.1" 404 1576 "" "AppleNewsBot" 1.99
2021-07-08T15:44:17.964002388Z
2021-07-08T15:44:17.964007158Z ==> /var/log/gitlab/gitlab-rails/production_json.log <==
2021-07-08T15:44:17.964012918Z {"method":"GET","path":"/-/acme-challenge","format":"html","controller":"AcmeChallengesController","action":"show","status":404,"time":"2021-07-08T15:44:17.812Z","params":[{"key":"domain","value":"jeffreyfreeman.me"},{"key":"token","value":"[FILTERED]"}],"remote_ip":null,"user_id":null,"username":null,"ua":null,"correlation_id":"CFKuNzhCjB9","cpu_s":0.01,"db_duration_s":0.00235,"view_duration_s":0.0,"duration_s":0.00842,"db_count":1,"db_write_count":0,"db_cached_count":0}
2021-07-08T15:44:19.034168993Z
2021-07-08T15:44:19.034195703Z ==> /var/log/gitlab/gitlab-pages/current <==
2021-07-08T15:44:19.034205654Z {"content_type":"text/html; charset=utf-8","correlation_id":"","duration_ms":4,"host":"jeffreyfreeman.me","level":"info","method":"GET","msg":"access","pages_host":"jeffreyfreeman.me","pages_https":false,"pages_project_id":31,"pages_project_prefix":"/","pages_project_serving_type":"file","proto":"HTTP/1.1","referrer":"","remote_addr":"192.168.0.4:50308","remote_ip":"17.58.92.204","status":307,"system":"http","time":"2021-07-08T15:44:18Z","ttfb_ms":4,"uri":"/.well-known/acme-challenge/F_uIkErnRk-jWtwbYEvotmq5yoGWtwH1zI7C_vKkHOc%22,%22remote_addr%22:%2252.28.236.88:0%22,%22remote_ip%22:%2252.28.236.88%22,%22status%22:404,%22system%22:%22http%22,%22time%22:%222021-01-20T17:30:12Z%22,%22uri%22:%22/-/acme-challenge?domain=jeffreyfreeman.me%5Cu0026token=%5BFILTERED%5D%22,%22user_agent%22:%22Mozilla/5.0","user_agent":"AppleNewsBot","written_bytes":123}
2021-07-08T15:44:19.034220074Z
2021-07-08T15:44:19.034224914Z ==> /var/log/gitlab/gitlab-workhorse/current <==
2021-07-08T15:44:19.034229964Z {"content_type":"","correlation_id":"6fQHOIDKD58","duration_ms":2,"host":"git.qoto.org","level":"info","method":"POST","msg":"access","proto":"HTTP/1.1","referrer":"","remote_addr":"18.232.151.21:0","remote_ip":"18.232.151.21","status":204,"system":"http","time":"2021-07-08T15:44:18Z","uri":"/api/v4/jobs/request","user_agent":"gitlab-runner 13.5.0 (13-5-stable; go1.13.8; linux/amd64)","written_bytes":0}
2021-07-08T15:44:19.034238764Z {"content_type":"text/html; charset=utf-8","correlation_id":"b58OHqTkMea","duration_ms":42,"host":"git.qoto.org","level":"info","method":"GET","msg":"access","proto":"HTTP/1.1","referrer":"","remote_addr":"17.58.92.204:0","remote_ip":"17.58.92.204","status":404,"system":"http","time":"2021-07-08T15:44:18Z","uri":"/-/acme-challenge?domain=jeffreyfreeman.me%5Cu0026token=pKAsjNmT6hfhhR1-Kv0xcWhFv_y9GseqxxRv4sgAi1M","user_agent":"AppleNewsBot","written_bytes":3108}
2021-07-08T15:44:19.034249305Z {"content_type":"text/html; charset=utf-8","correlation_id":"0YdnOnfN0q9","duration_ms":59,"host":"git.qoto.org","level":"info","method":"GET","msg":"access","proto":"HTTP/1.1","referrer":"","remote_addr":"17.58.92.204:0","remote_ip":"17.58.92.204","status":404,"system":"http","time":"2021-07-08T15:44:18Z","uri":"/-/acme-challenge?domain=jeffreyfreeman.me\u0026token=[FILTERED]","user_agent":"AppleNewsBot","written_bytes":3108}
2021-07-08T15:44:19.034258695Z
2021-07-08T15:44:19.034262905Z ==> /var/log/gitlab/nginx/gitlab_access.log <==
2021-07-08T15:44:19.034285846Z 192.168.0.5 - - [08/Jul/2021:15:44:18 +0000] "POST /api/v4/jobs/request HTTP/1.1" 204 0 "" "gitlab-runner 13.5.0 (13-5-stable; go1.13.8; linux/amd64)" -
2021-07-08T15:44:19.034292196Z 192.168.0.6 - - [08/Jul/2021:15:44:18 +0000] "GET /-/acme-challenge?domain=jeffreyfreeman.me%5Cu0026token=pKAsjNmT6hfhhR1-Kv0xcWhFv_y9GseqxxRv4sgAi1M HTTP/1.1" 404 1576 "" "AppleNewsBot" 1.99
2021-07-08T15:44:19.034297966Z 192.168.0.6 - - [08/Jul/2021:15:44:18 +0000] "GET /-/acme-challenge?domain=jeffreyfreeman.me&token=acme-challenge HTTP/1.1" 404 1576 "" "AppleNewsBot" 1.99
2021-07-08T15:44:19.034303646Z
2021-07-08T15:44:19.034307886Z ==> /var/log/gitlab/gitlab-rails/production_json.log <==
2021-07-08T15:44:19.034312896Z {"method":"GET","path":"/-/acme-challenge","format":"html","controller":"AcmeChallengesController","action":"show","status":404,"time":"2021-07-08T15:44:18.121Z","params":[{"key":"domain","value":"jeffreyfreeman.me\\u0026token=pKAsjNmT6hfhhR1-Kv0xcWhFv_y9GseqxxRv4sgAi1M"}],"remote_ip":null,"user_id":null,"username":null,"ua":null,"correlation_id":"b58OHqTkMea","cpu_s":0.01,"db_duration_s":0.00088,"view_duration_s":0.0,"duration_s":0.00394,"db_count":1,"db_write_count":0,"db_cached_count":0}
2021-07-08T15:44:19.034323017Z {"method":"GET","path":"/-/acme-challenge","format":"html","controller":"AcmeChallengesController","action":"show","status":404,"time":"2021-07-08T15:44:18.877Z","params":[{"key":"domain","value":"jeffreyfreeman.me"},{"key":"token","value":"[FILTERED]"}],"remote_ip":null,"user_id":null,"username":null,"ua":null,"correlation_id":"0YdnOnfN0q9","cpu_s":0.01,"db_duration_s":0.00523,"view_duration_s":0.0,"duration_s":0.0258,"db_count":1,"db_write_count":0,"db_cached_count":0}
2021-07-08T15:44:19.034332747Z
2021-07-08T15:44:19.034336897Z ==> /var/log/gitlab/gitlab-rails/production.log <==
2021-07-08T15:44:19.034341657Z Started GET "/-/acme-challenge?domain=jeffreyfreeman.me%5Cu0026token=pKAsjNmT6hfhhR1-Kv0xcWhFv_y9GseqxxRv4sgAi1M" for 17.58.92.204 at 2021-07-08 15:44:18 +0000
2021-07-08T15:44:19.122782491Z Processing by AcmeChallengesController#show as HTML
2021-07-08T15:44:19.122809042Z Parameters: {"domain"=>"jeffreyfreeman.me\\u0026token=pKAsjNmT6hfhhR1-Kv0xcWhFv_y9GseqxxRv4sgAi1M"}
2021-07-08T15:44:19.122816222Z Completed 404 Not Found in 4ms (ActiveRecord: 0.9ms | Elasticsearch: 0.0ms | Allocations: 469)
2021-07-08T15:44:19.122821582Z Started GET "/-/acme-challenge?domain=jeffreyfreeman.me&token=[FILTERED]" for 17.58.92.204 at 2021-07-08 15:44:18 +0000
2021-07-08T15:44:19.122827582Z Processing by AcmeChallengesController#show as HTML
2021-07-08T15:44:19.122832552Z Parameters: {"domain"=>"jeffreyfreeman.me", "token"=>"[FILTERED]"}
2021-07-08T15:44:19.122838713Z Completed 404 Not Found in 26ms (ActiveRecord: 5.2ms | Elasticsearch: 0.0ms | Allocations: 479)
2021-07-08T15:44:19.122843613Z
2021-07-08T15:44:19.122847963Z ==> /var/log/gitlab/gitlab-pages/current <==
2021-07-08T15:44:19.122853053Z {"content_type":"text/html; charset=utf-8","correlation_id":"","duration_ms":0,"host":"jeffreyfreeman.me","level":"info","method":"GET","msg":"access","pages_host":"jeffreyfreeman.me","pages_https":false,"pages_project_id":31,"pages_project_prefix":"/","pages_project_serving_type":"file","proto":"HTTP/1.1","referrer":"","remote_addr":"192.168.0.4:50312","remote_ip":"17.58.92.204","status":307,"system":"http","time":"2021-07-08T15:44:18Z","ttfb_ms":0,"uri":"/.well-known/acme-challenge/F_uIkErnRk-jWtwbYEvotmq5yoGWtwH1zI7C_vKkHOc","user_agent":"AppleNewsBot","written_bytes":152}
2021-07-08T15:44:20.152084487Z {"content_type":"text/html; charset=utf-8","correlation_id":"","duration_ms":0,"host":"jeffreyfreeman.me","level":"info","method":"GET","msg":"access","pages_host":"jeffreyfreeman.me","pages_https":false,"pages_project_id":31,"pages_project_prefix":"/","pages_project_serving_type":"file","proto":"HTTP/1.1","referrer":"","remote_addr":"192.168.0.4:50314","remote_ip":"17.58.92.204","status":307,"system":"http","time":"2021-07-08T15:44:19Z","ttfb_ms":0,"uri":"/.well-known/acme-challenge/F_uIkErnRk-jWtwbYEvotmq5yoGWtwH1zI7C_vKkHOc%22,%22remote_addr%22:%2264.78.149.164:0%22,%22remote_ip%22:%2264.78.149.164%22,%22status%22:404,%22system%22:%22http%22,%22time%22:%222021-01-20T17:30:12Z%22,%22uri%22:%22/-/acme-challenge?domain=jeffreyfreeman.me%5Cu0026token=%5BFILTERED%5D%22,%22user_agent%22:%22Mozilla/5.0","user_agent":"AppleNewsBot","written_bytes":123}
2021-07-08T15:44:20.152117908Z {"content_type":"text/html; charset=utf-8","correlation_id":"","duration_ms":0,"host":"jeffreyfreeman.me","level":"info","method":"GET","msg":"access","pages_host":"jeffreyfreeman.me","pages_https":false,"pages_project_id":31,"pages_project_prefix":"/","pages_project_serving_type":"file","proto":"HTTP/1.1","referrer":"","remote_addr":"192.168.0.4:50316","remote_ip":"17.58.92.204","status":307,"system":"http","time":"2021-07-08T15:44:19Z","ttfb_ms":0,"uri":"/.well-known/acme-challenge/F_uIkErnRk-jWtwbYEvotmq5yoGWtwH1zI7C_vKkHOc","user_agent":"AppleNewsBot","written_bytes":152}
2021-07-08T15:44:20.152130748Z
2021-07-08T15:44:20.152136009Z ==> /var/log/gitlab/gitlab-workhorse/current <==
2021-07-08T15:44:20.152141939Z {"content_type":"","correlation_id":"YgLwFaQQLo1","duration_ms":0,"host":"git.qoto.org","level":"info","method":"POST","msg":"access","proto":"HTTP/1.1","referrer":"","remote_addr":"18.232.151.21:0","remote_ip":"18.232.151.21","status":204,"system":"http","time":"2021-07-08T15:44:19Z","uri":"/api/v4/jobs/request","user_agent":"gitlab-runner 13.5.0 (13-5-stable; go1.13.8; linux/amd64)","written_bytes":0}
2021-07-08T15:44:20.152153149Z {"content_type":"text/html; charset=utf-8","correlation_id":"4UTanvbymF5","duration_ms":14,"host":"git.qoto.org","level":"info","method":"GET","msg":"access","proto":"HTTP/1.1","referrer":"","remote_addr":"17.58.92.204:0","remote_ip":"17.58.92.204","status":404,"system":"http","time":"2021-07-08T15:44:19Z","uri":"/-/acme-challenge?domain=jeffreyfreeman.me\u0026token=[FILTERED]","user_agent":"AppleNewsBot","written_bytes":3108}
2021-07-08T15:44:20.152162729Z {"content_type":"text/html; charset=utf-8","correlation_id":"d3L7oyTQXq","duration_ms":14,"host":"git.qoto.org","level":"info","method":"GET","msg":"access","proto":"HTTP/1.1","referrer":"","remote_addr":"17.58.92.204:0","remote_ip":"17.58.92.204","status":404,"system":"http","time":"2021-07-08T15:44:19Z","uri":"/-/acme-challenge?domain=jeffreyfreeman.me\u0026token=[FILTERED]","user_agent":"AppleNewsBot","written_bytes":3108}
2021-07-08T15:44:20.152188870Z {"content_type":"text/html; charset=utf-8","correlation_id":"W9Xb3iKsYD3","duration_ms":11,"host":"git.qoto.org","level":"info","method":"GET","msg":"access","proto":"HTTP/1.1","referrer":"","remote_addr":"17.58.92.204:0","remote_ip":"17.58.92.204","status":404,"system":"http","time":"2021-07-08T15:44:19Z","uri":"/-/acme-challenge?domain=jeffreyfreeman.me\u0026token=[FILTERED]","user_agent":"AppleNewsBot","written_bytes":3108}
2021-07-08T15:44:20.152198760Z {"content_type":"","correlation_id":"ybtlPyCEqM5","duration_ms":3,"host":"git.qoto.org","level":"info","method":"POST","msg":"access","proto":"HTTP/1.1","referrer":"","remote_addr":"18.232.151.21:0","remote_ip":"18.232.151.21","status":204,"system":"http","time":"2021-07-08T15:44:20Z","uri":"/api/v4/jobs/request","user_agent":"gitlab-runner 13.5.0 (13-5-stable; go1.13.8; linux/amd64)","written_bytes":0}
2021-07-08T15:44:20.152207680Z
2021-07-08T15:44:20.152212221Z ==> /var/log/gitlab/nginx/gitlab_access.log <==
2021-07-08T15:44:20.152217091Z 192.168.0.3 - - [08/Jul/2021:15:44:19 +0000] "POST /api/v4/jobs/request HTTP/1.1" 204 0 "" "gitlab-runner 13.5.0 (13-5-stable; go1.13.8; linux/amd64)" -
2021-07-08T15:44:20.152222261Z 192.168.0.6 - - [08/Jul/2021:15:44:19 +0000] "GET /-/acme-challenge?domain=jeffreyfreeman.me&token=F_uIkErnRk-jWtwbYEvotmq5yoGWtwH1zI7C_vKkHOc HTTP/1.1" 404 1576 "" "AppleNewsBot" 1.99
2021-07-08T15:44:20.295562757Z 192.168.0.6 - - [08/Jul/2021:15:44:19 +0000] "GET /-/acme-challenge?domain=jeffreyfreeman.me&token=acme-challenge HTTP/1.1" 404 1576 "" "AppleNewsBot" 1.99
2021-07-08T15:44:20.295590007Z 192.168.0.6 - - [08/Jul/2021:15:44:19 +0000] "GET /-/acme-challenge?domain=jeffreyfreeman.me&token=F_uIkErnRk-jWtwbYEvotmq5yoGWtwH1zI7C_vKkHOc HTTP/1.1" 404 1576 "" "AppleNewsBot" 1.99
2021-07-08T15:44:20.295596577Z 192.168.0.4 - - [08/Jul/2021:15:44:20 +0000] "POST /api/v4/jobs/request HTTP/1.1" 204 0 "" "gitlab-runner 13.5.0 (13-5-stable; go1.13.8; linux/amd64)" -
2021-07-08T15:44:20.295602478Z
2021-07-08T15:44:20.295607258Z ==> /var/log/gitlab/gitlab-rails/production_json.log <==
2021-07-08T15:44:20.295612228Z {"method":"GET","path":"/-/acme-challenge","format":"html","controller":"AcmeChallengesController","action":"show","status":404,"time":"2021-07-08T15:44:19.246Z","params":[{"key":"domain","value":"jeffreyfreeman.me"},{"key":"token","value":"[FILTERED]"}],"remote_ip":null,"user_id":null,"username":null,"ua":null,"correlation_id":"4UTanvbymF5","cpu_s":0.01,"db_duration_s":0.00083,"view_duration_s":0.0,"duration_s":0.00365,"db_count":1,"db_write_count":0,"db_cached_count":0}
2021-07-08T15:44:20.295623698Z {"method":"GET","path":"/-/acme-challenge","format":"html","controller":"AcmeChallengesController","action":"show","status":404,"time":"2021-07-08T15:44:19.600Z","params":[{"key":"domain","value":"jeffreyfreeman.me"},{"key":"token","value":"[FILTERED]"}],"remote_ip":null,"user_id":null,"username":null,"ua":null,"correlation_id":"d3L7oyTQXq","cpu_s":0.01,"db_duration_s":0.0011,"view_duration_s":0.0,"duration_s":0.00382,"db_count":1,"db_write_count":0,"db_cached_count":0}
2021-07-08T15:44:20.295662459Z {"method":"GET","path":"/-/acme-challenge","format":"html","controller":"AcmeChallengesController","action":"show","status":404,"time":"2021-07-08T15:44:19.949Z","params":[{"key":"domain","value":"jeffreyfreeman.me"},{"key":"token","value":"[FILTERED]"}],"remote_ip":null,"user_id":null,"username":null,"ua":null,"correlation_id":"W9Xb3iKsYD3","cpu_s":0.01,"db_duration_s":0.00087,"view_duration_s":0.0,"duration_s":0.00346,"db_count":1,"db_write_count":0,"db_cached_count":0}
2021-07-08T15:44:20.295675569Z
2021-07-08T15:44:20.295680340Z ==> /var/log/gitlab/gitlab-rails/production.log <==
2021-07-08T15:44:20.295685550Z Started GET "/-/acme-challenge?domain=jeffreyfreeman.me&token=[FILTERED]" for 17.58.92.204 at 2021-07-08 15:44:19 +0000
2021-07-08T15:44:20.295692630Z Processing by AcmeChallengesController#show as HTML
2021-07-08T15:44:20.295697590Z Parameters: {"domain"=>"jeffreyfreeman.me", "token"=>"[FILTERED]"}
2021-07-08T15:44:20.295702800Z Completed 404 Not Found in 4ms (ActiveRecord: 0.8ms | Elasticsearch: 0.0ms | Allocations: 470)
2021-07-08T15:44:20.295707570Z Started GET "/-/acme-challenge?domain=jeffreyfreeman.me&token=[FILTERED]" for 17.58.92.204 at 2021-07-08 15:44:19 +0000
2021-07-08T15:44:20.295712620Z Processing by AcmeChallengesController#show as HTML
2021-07-08T15:44:20.295717270Z Parameters: {"domain"=>"jeffreyfreeman.me", "token"=>"[FILTERED]"}
2021-07-08T15:44:20.295722811Z Completed 404 Not Found in 4ms (ActiveRecord: 1.1ms | Elasticsearch: 0.0ms | Allocations: 470)
2021-07-08T15:44:20.295727361Z Started GET "/-/acme-challenge?domain=jeffreyfreeman.me&token=[FILTERED]" for 17.58.92.204 at 2021-07-08 15:44:19 +0000
2021-07-08T15:44:20.295732271Z Processing by AcmeChallengesController#show as HTML
2021-07-08T15:44:20.295736951Z Parameters: {"domain"=>"jeffreyfreeman.me", "token"=>"[FILTERED]"}
2021-07-08T15:44:20.295741921Z Completed 404 Not Found in 3ms (ActiveRecord: 0.9ms | Elasticsearch: 0.0ms | Allocations: 470)
2021-07-08T15:44:20.295746661Z Started GET "/-/acme-challenge?domain=jeffreyfreeman.me&token=[FILTERED]" for 17.58.92.204 at 2021-07-08 15:44:20 +0000
2021-07-08T15:44:20.295751491Z Processing by AcmeChallengesController#show as HTML
2021-07-08T15:44:20.295756191Z Parameters: {"domain"=>"jeffreyfreeman.me", "token"=>"[FILTERED]"}
2021-07-08T15:44:20.295761552Z Completed 404 Not Found in 4ms (ActiveRecord: 0.9ms | Elasticsearch: 0.0ms | Allocations: 470)
2021-07-08T15:44:20.295766292Z
2021-07-08T15:44:20.295770882Z ==> /var/log/gitlab/gitlab-workhorse/current <==
2021-07-08T15:44:20.295776092Z {"content_type":"text/html; charset=utf-8","correlation_id":"p1wP9F5Yg14","duration_ms":16,"host":"git.qoto.org","level":"info","method":"GET","msg":"access","proto":"HTTP/1.1","referrer":"","remote_addr":"17.58.92.204:0","remote_ip":"17.58.92.204","status":404,"system":"http","time":"2021-07-08T15:44:20Z","uri":"/-/acme-challenge?domain=jeffreyfreeman.me\u0026token=[FILTERED]","user_agent":"AppleNewsBot","written_bytes":3108}
2021-07-08T15:44:20.295793162Z
2021-07-08T15:44:20.295797633Z ==> /var/log/gitlab/nginx/gitlab_access.log <==
2021-07-08T15:44:20.295802343Z 192.168.0.6 - - [08/Jul/2021:15:44:20 +0000] "GET /-/acme-challenge?domain=jeffreyfreeman.me&token=F_uIkErnRk-jWtwbYEvotmq5yoGWtwH1zI7C_vKkHOc HTTP/1.1" 404 1576 "" "AppleNewsBot" 1.99
2021-07-08T15:44:20.295808143Z
2021-07-08T15:44:20.295812413Z ==> /var/log/gitlab/gitlab-rails/production_json.log <==
2021-07-08T15:44:20.335388351Z {"method":"GET","path":"/-/acme-challenge","format":"html","controller":"AcmeChallengesController","action":"show","status":404,"time":"2021-07-08T15:44:20.217Z","params":[{"key":"domain","value":"jeffreyfreeman.me"},{"key":"token","value":"[FILTERED]"}],"remote_ip":null,"user_id":null,"username":null,"ua":null,"correlation_id":"p1wP9F5Yg14","cpu_s":0.01,"db_duration_s":0.00085,"view_duration_s":0.0,"duration_s":0.00395,"db_count":1,"db_write_count":0,"db_cached_count":0}
2021-07-08T15:44:21.413783281Z
2021-07-08T15:44:21.413807362Z ==> /var/log/gitlab/gitlab-workhorse/current <==
2021-07-08T15:44:21.413814162Z {"content_type":"text/html; charset=utf-8","correlation_id":"VtAdxizG285","duration_ms":13,"host":"git.qoto.org","level":"info","method":"GET","msg":"access","proto":"HTTP/1.1","referrer":"","remote_addr":"17.58.92.204:0","remote_ip":"17.58.92.204","status":404,"system":"http","time":"2021-07-08T15:44:20Z","uri":"/-/acme-challenge?domain=jeffreyfreeman.me%5Cu0026token=F_uIkErnRk-jWtwbYEvotmq5yoGWtwH1zI7C_vKkHOc","user_agent":"AppleNewsBot","written_bytes":3108}
2021-07-08T15:44:21.413825842Z {"content_type":"","correlation_id":"MiodSYKVsOa","duration_ms":0,"host":"git.qoto.org","level":"info","method":"POST","msg":"access","proto":"HTTP/1.1","referrer":"","remote_addr":"18.232.151.21:0","remote_ip":"18.232.151.21","status":204,"system":"http","time":"2021-07-08T15:44:21Z","uri":"/api/v4/jobs/request","user_agent":"gitlab-runner 13.5.0 (13-5-stable; go1.13.8; linux/amd64)","written_bytes":0}
2021-07-08T15:44:21.413835693Z
2021-07-08T15:44:21.413840993Z ==> /var/log/gitlab/nginx/gitlab_access.log <==
2021-07-08T15:44:21.413846413Z 192.168.0.6 - - [08/Jul/2021:15:44:20 +0000] "GET /-/acme-challenge?domain=jeffreyfreeman.me%5Cu0026token=F_uIkErnRk-jWtwbYEvotmq5yoGWtwH1zI7C_vKkHOc HTTP/1.1" 404 1576 "" "AppleNewsBot" 1.99
2021-07-08T15:44:21.413851103Z 192.168.0.5 - - [08/Jul/2021:15:44:21 +0000] "POST /api/v4/jobs/request HTTP/1.1" 204 0 "" "gitlab-runner 13.5.0 (13-5-stable; go1.13.8; linux/amd64)" -
2021-07-08T15:44:21.413854783Z
2021-07-08T15:44:21.413857663Z ==> /var/log/gitlab/gitlab-rails/production_json.log <==
2021-07-08T15:44:21.413861023Z {"method":"GET","path":"/-/acme-challenge","format":"html","controller":"AcmeChallengesController","action":"show","status":404,"time":"2021-07-08T15:44:20.482Z","params":[{"key":"domain","value":"jeffreyfreeman.me\\u0026token=F_uIkErnRk-jWtwbYEvotmq5yoGWtwH1zI7C_vKkHOc"}],"remote_ip":null,"user_id":null,"username":null,"ua":null,"correlation_id":"VtAdxizG285","cpu_s":0.01,"db_duration_s":0.00097,"view_duration_s":0.0,"duration_s":0.0038,"db_count":1,"db_write_count":0,"db_cached_count":0}
2021-07-08T15:44:21.413883954Z
2021-07-08T15:44:21.413887204Z ==> /var/log/gitlab/gitlab-rails/production.log <==
2021-07-08T15:44:21.413891464Z Started GET "/-/acme-challenge?domain=jeffreyfreeman.me%5Cu0026token=F_uIkErnRk-jWtwbYEvotmq5yoGWtwH1zI7C_vKkHOc" for 17.58.92.204 at 2021-07-08 15:44:20 +0000
2021-07-08T15:44:21.413895024Z Processing by AcmeChallengesController#show as HTML
2021-07-08T15:44:21.413898044Z Parameters: {"domain"=>"jeffreyfreeman.me\\u0026token=F_uIkErnRk-jWtwbYEvotmq5yoGWtwH1zI7C_vKkHOc"}
2021-07-08T15:44:21.413902004Z Completed 404 Not Found in 4ms (ActiveRecord: 1.0ms | Elasticsearch: 0.0ms | Allocations: 469)
I deleted the pages configuration entirely for that site then recreated it. I had to re verify my domain but once I did it was able to get a LE certificate just fine. Must be some sort of bug in gitlab I guess.