For the last few weeks, I have repeatedly attempted to enable SSL for a custom domain with Gitlab Pages (on gitlab.com), but the SSL certificate creation always fails. The custom domain is verified and HTTP access works, but the Let’s Encrypt SSL certificate creation won’t succeed. Among other things, I have clicked the “Retry” button on the custom domain record after the creation fails.
I have also tried numerous times deleting the custom domain record in the Gitlab UI and going through the entire DNS and verification flow again (updating the TXT record with the new verification value each time), but that hasn’t done anything – the certificate creation always fails.
Another way to test and verify - install certbot locally, and request certificates to see whether it works. Something like certbot certonly --standalone -d yourdomain.com --staple-ocsp -m test@yourdomain.com--agree-tos --preferred-challenges dns, using the ACME-DNS challenge.
$ sudo certbot certonly --standalone -d palousedata.co --staple-ocsp -m test@palousedata.co --agree-tos --preferred-challenges dns
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Requesting a certificate for palousedata.co
None of the preferred challenges are supported by the selected plugin
Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for more details.
Interesting. I still get SERVFAIL which could mean that the DNS servers are not configured correctly - i.e. the DNS server where your resolval queries are answered, is correct, but mine is not.
Tried using 8.8.8.8 or 1.1.1.1 as resolvers, still failing for different record types (A, NS, etc.).
If I ask the nameservers if they know the domain, or feel authoritative for the domain, these are the results. Looks normal.
dig palousedata.co A @1-you.njalla.no ─╯
; <<>> DiG 9.10.6 <<>> palousedata.co A @1-you.njalla.no
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 60388
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;palousedata.co. IN A
;; ANSWER SECTION:
palousedata.co. 10800 IN A 35.185.44.232
;; Query time: 38 msec
;; SERVER: 2001:67c:235c::2#53(2001:67c:235c::2)
;; WHEN: Tue Oct 17 11:17:53 CEST 2023
;; MSG SIZE rcvd: 59
dig palousedata.co A @2-can.njalla.in ─╯
; <<>> DiG 9.10.6 <<>> palousedata.co A @2-can.njalla.in
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 12244
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;palousedata.co. IN A
;; ANSWER SECTION:
palousedata.co. 10800 IN A 35.185.44.232
;; Query time: 38 msec
;; SERVER: 2001:67c:235c::34#53(2001:67c:235c::34)
;; WHEN: Tue Oct 17 11:18:44 CEST 2023
;; MSG SIZE rcvd: 59
dig palousedata.co A @3-get.njalla.fo ─╯
; <<>> DiG 9.10.6 <<>> palousedata.co A @3-get.njalla.fo
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 32294
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;palousedata.co. IN A
;; ANSWER SECTION:
palousedata.co. 10800 IN A 35.185.44.232
;; Query time: 35 msec
;; SERVER: 2001:67c:2354:2::5#53(2001:67c:2354:2::5)
;; WHEN: Tue Oct 17 11:19:03 CEST 2023
;; MSG SIZE rcvd: 59
Which leads me to the assumption that the nameservers are configured to serve responses for the domain, but the upper resolval chain is somehow broken.
Doing a DNS trace to lookup the nameserver tree also looks fine.
dig palousedata.co +trace ─╯
; <<>> DiG 9.10.6 <<>> palousedata.co +trace
;; global options: +cmd
. 465669 IN NS h.root-servers.net.
. 465669 IN NS j.root-servers.net.
. 465669 IN NS g.root-servers.net.
. 465669 IN NS d.root-servers.net.
. 465669 IN NS k.root-servers.net.
. 465669 IN NS c.root-servers.net.
. 465669 IN NS l.root-servers.net.
. 465669 IN NS a.root-servers.net.
. 465669 IN NS f.root-servers.net.
. 465669 IN NS i.root-servers.net.
. 465669 IN NS b.root-servers.net.
. 465669 IN NS e.root-servers.net.
. 465669 IN NS m.root-servers.net.
. 465669 IN RRSIG NS 8 0 518400 20231030040000 20231017030000 46780 . J1f+Yl/fRnNbk4WuLUz7kVd0xQD08y/ELnjDiC72UpUqN9Ia0YLus8bP GeJzJOcSjbqXf9nW9Sp5m4V6QljcoXsbZeJ1otgRlqFNTTkFYNJIR86H s7U9k8RHtI7da1ebxAROl4m4EmbIYTTDrpDqDiCtoUtw4GEfKh3xorOd OdE5f2DEOksLwc17l7gf0BwFZoSRnW+FNRvAW7GGiYs0tb2L/e+DNHyR nZyH24JUK28d3vUSF3mUG7KK7VRRCjlVuEciojpuNbcPcPs+KyjRwhu8 eTZrTKMhgIo7vpmVkQBuvqVjY/GwL7Id2c2afWvHagM+wENyxyr4vY1f tPy1fg==
;; Received 525 bytes from fe80::1%14#53(fe80::1%14) in 40 ms
co. 172800 IN NS ns2.cctld.co.
co. 172800 IN NS ns3.cctld.co.
co. 172800 IN NS ns6.cctld.co.
co. 172800 IN NS ns5.cctld.co.
co. 172800 IN NS ns1.cctld.co.
co. 172800 IN NS ns4.cctld.co.
co. 86400 IN DS 62110 8 2 4857EE1A85E661C92156AD09B40139DAE7D5BE37D117A1223289F0E2 0CF9D654
co. 86400 IN DS 61744 8 2 B9ADCA87453C1F69A9A524681F4179A88374C546A45892ACC10E2653 519ACCBD
co. 86400 IN RRSIG DS 8 1 86400 20231030050000 20231017040000 46780 . uicvK2LRlazKqMhVnbTqmJ97FxIYsDZuKf8tzc9sIABuXOdYaLzKB4Av rmlAT3vSqy9fBjo40M52wvOxfRs4BAy7M38JbXGTuprQVo4G/zxKuQq2 +GVGgzuvoXOApmujOuGUZCjQFBbmI7sAvhIv+iJYJCOJ2LKC4H90TDBK DTddZu9KtU4E9vz31X+YoNe06E9thmNX+4J4L3M39Bhm8dicfWiCPIwQ W0tDDHZTSdftwDtqxTt7d/4EujrQyzf1kxRdh8XrNi01eWlNmRYTxauA VRCW8p+kPrVgEtzw+KyX0EQ+sIMIWtsBYkUBSeULIixrAKHzgxjX5kyK ZsvW1A==
;; Received 806 bytes from 193.0.14.129#53(k.root-servers.net) in 38 ms
palousedata.co. 3600 IN NS 3-get.njalla.fo.
palousedata.co. 3600 IN NS 1-you.njalla.no.
palousedata.co. 3600 IN NS 2-can.njalla.in.
palousedata.co. 3600 IN DS 39709 8 2 B6BBE6BC938D63218F7C3091C56DB67B656B37A6386848B36B013501 A52BC6BF
palousedata.co. 3600 IN RRSIG DS 8 2 3600 20231112142228 20231013140710 19065 co. Qu+zWTLfvRDZvjItKoXnZz81Ru8YVywiA1XnGYO1guc9WqHAC/fAx6gw LYOtU3AblvhAPgd/uhBdwtuI0U1AD0UEJdeaSZckqDHZELkYbcCytjQw +EI07ix5oVHEpIdSUsblU859NfBM+LTH+oYMouoHUG2IYBTh+cdHodqL LfJ5aafFERIplBnqLy7of0xjbp8e9ZPxkRRo9CeZWF4qUA==
;; Received 372 bytes from 156.154.104.25#53(ns5.cctld.co) in 23 ms
palousedata.co. 10800 IN A 35.185.44.232
palousedata.co. 10800 IN RRSIG A 8 2 10800 20240110094454 20231011094454 35670 palousedata.co. V8F+emZcZgiBnVe9XuqrLgQD+D3U7n9QBbVyT1RfSxpMrb1/zCZrnXDE IYBt60JKMSs7mOBJW7jIAcIEm0dsLXAFmKUYubp/ja/nffbtFG0Uj/6r qOzPbYEeHtwmF5Pze43RqxIcVZR67gBTPpM4WfPbL34eAYXQ72o2xfPC u7M61tMdz3aGFAqYQ3C4f9M6wBgz9R8h16bbTnqts4yXUT8KHfq9oYxP 7CCvUHfBcJgo2YLq7txDiMpV7r9eRxzmAKsmcrLykoDLDwQduj2jhgJm ZdcCZPNfDNWGxJIG1XDKFGZo2t8dmr14M8MMSEQNwvi2eQFgoes56Wif TR3eDg==
palousedata.co. 10800 IN RRSIG A 8 2 10800 20240110094454 20231011094454 60093 palousedata.co. cc7cNN0YlF2ZYPOFsqP6MCVFB+9ifEpzqkHyQC0+70sUyv9eL7Plbbrr kgJAWG3A4sgGLbvOfax2o85uDvvRMR3uHnGkDLiZvQqnshDPC6zVsW1z FJvNKuMsAPlILwDei8ZVAye6jJOOaULBsyFhRPdlHoLyNGluN8N4O5dx QtZ+SUIifgMIhzfrxkFboc6RgRVFXaQMpj/LgK4yMo6aFmWonyVqfsLh olJVWP+V75OYsEfgK+AQmFPrtldiy78F+cC8v7+KHcaNoWICP1yxNjCd 5fB59MUtbLhQhQCzKcEzQyF3tOpOUBU9imo4gXrQh0P0U9TtjkOrfsPh 9NMTlg==
palousedata.co. 36000 IN NS 1-you.njalla.no.
palousedata.co. 36000 IN NS 2-can.njalla.in.
palousedata.co. 36000 IN NS 3-get.njalla.fo.
palousedata.co. 36000 IN RRSIG NS 8 2 36000 20240110094454 20231011094454 35670 palousedata.co. VAT482qRkZPawaCgO7Z+O6mHLWyVpeB1kCZRMfbbcbmjr2D2FLu6OFkA npPlZGkwU4BIDvFXKDeIlQIwLD10rO11aT6Q9uCJIpzpXUiSxMRYIv7l SSdPXoAqF1lx3H1pdVnIstCWKCo81ZRJDli6TWwWLBuNnNL/dCo1LvQp mlVlAMJb100sWEarDudhZusBcUh2B4olYLalzkdSc1O1FJpS43p7+Jh1 PJ1rflNU9cQq6wabYNEJGj5gbzfb/L+FpWS07ix0HeCgj8V/f8LTjkO4 Qkg7gOkawArQmzdaqXclzWBluWwyfNwvXIzUzbRiYuU2Zan9Av1MR8S2 lpUcbA==
palousedata.co. 36000 IN RRSIG NS 8 2 36000 20240110094454 20231011094454 60093 palousedata.co. W31d2RKJiOaCmgKSsMRdndaeR3etUY5Jpmd82yjv5lqFFijg3ewzX6Tl qB4qpJe9Qw7ESreLAuOGNT2/QVsX1pXYTRy3HihwtAEpQBKaT1ckoo6g ARmOX4mLJSOT4IL1s5AqbOMy9w5iBaGTh7lZ3tLp+z9Yr6WGCNvebfMZ 3Ksg88h1cOZgLRMk+1ApSQJuTezjbz7fA7W7bAuVrreOtE9ps0JxuLNq eVnwbdU76mmRGlJtA3fcY4/oLEgz+VTCnO5CGQafiRILoPPJW8SPgAQZ TL/AoCyeU7ZJnqdToPGmAAzB3cvZu53hAxM5IcV74opNeu3lrzQMLpGb V1wkNw==
;; Received 1354 bytes from 95.215.19.5#53(3-get.njalla.fo) in 51 ms
Different host
To verify that is not my local DTAG ASN causing this error, I ran the queries on a VM in Hetzner Cloud - same results, DNS records cannot be resolved and return SERVFAIL.
The caching server encountered NXDOMAIN or SERVFAIL when chasing glue records to find an authoritative nameserver, and this event has been cached. Even if the problem has been corrected, or the glue has been pointed somewhere else, the nameserver isn’t going to try asking for it again until an internal timer expires. Requesting a cache purge for the zone in question will usually reset it.
All resolvers have cached the wrong nameserver settings, and as such, continue returning SERVFAIL. That would explain the behavior I am seeing - and your resolver has cached a “good” result.
Looking into the 3 nameservers again, there are some odd configurations:
The first nameserver has A/AAAA records but no NS records. The SOA is interesting - ns1.njal.la as authortiative NS, and you.can-get-no.info as email address.
dig 1-you.njalla.no
;; ANSWER SECTION:
1-you.njalla.no. 6989 IN A 185.193.124.2
;; ADDITIONAL SECTION:
1-you.njalla.no. 6989 IN AAAA 2001:67c:235c::2
;; AUTHORITY SECTION:
njalla.no. 3402 IN SOA ns1.njal.la. you.can-get-no.info. 2310050945 21600 7200 1814400 3600
dig 2-can.njalla.in
;; ANSWER SECTION:
2-can.njalla.in. 2693 IN A 185.193.124.34
2-can.njalla.in. 5662 IN AAAA 2001:67c:235c::34
;; AUTHORITY SECTION:
njalla.in. 2693 IN NS ns1.njal.la.
njalla.in. 2693 IN NS ns3.njal.la.
njalla.in. 2693 IN NS ns2.njal.la.
dig 3-get.njalla.fo
;; ANSWER SECTION:
3-get.njalla.fo. 6676 IN A 95.215.19.5
;; AUTHORITY SECTION:
njalla.fo. 6676 IN NS ns1.njal.la.
njalla.fo. 6676 IN NS ns3.njal.la.
njalla.fo. 6676 IN NS ns2.njal.la.
njalla.fo. 6676 IN NS 3-get.njalla.fo.
njalla.fo. 3600 IN SOA ns1.njal.la. you.can-get-no.info. 2310130945 21600 7200 1814400 3600
;; ADDITIONAL SECTION:
3-get.njalla.fo. 6676 IN AAAA 2001:67c:2354:2::5
Reading dig output manually is hard. Changed the route and googled for opensrs njalla.fo servfail to see if someone else had the problem. No - but the website spotted my eye.
Suggest contacting your domain support about the missing glue records. This is something only their infrastructure/support team can debug and fix. Link them this forum topic for deeper analysis.
Here’s the transcript from my interchange with Njalla’s support. It was some kind of DNSSEC issue on their end, that, once they had fixed it, the Gitlab Pages SSL cert creation succeeded and I can now access my Gitlab page via HTTPS with my custom domain name.
Note: I had to put backticks around the links below due to a 10-links-per-message limit here on the forum. Sorry they’re not clickable.
Njalla support transcript
Title: Missing glue record even though I’m using Njalla’s nameservers
Hi there,
When using https://check.njal.la/dns/?name=palousedata.co to debug some DNS challenges that I recently had with the domain palousedata.co, the analysis is showing missing glue records. When I go to the palousedata.co domain configuration, it is setup to use Njalla’s nameservers, so my understanding from your FAQ (https://njal.la/faq/) is that I shouldn’t need to configure any glue records (and that they shouldn’t be missing).
For context, here’s the original issue that surfaced this finding (I’m the user whitecoop in the thread): https://forum.gitlab.com/t/gitlab-com-project-pages-ssl-certificate-creation-repeatedly-failing/93974?u=whitecoop
And here’s the specific message where the missing glue records were found: https://forum.gitlab.com/t/gitlab-com-project-pages-ssl-certificate-creation-repeatedly-failing/93974/4?u=whitecoop
I would appreciate any guidance you have. Thank you
Reply #1 (from Njalla)
Since you use our name servers there is no need for glue records
Reply #2 (from me)
So, the missing glue records shouldn’t have anything to do with the SERVFAILs that the gitlab.com troubleshooting got on multiple machines (in this message here: https://forum.gitlab.com/t/gitlab-com-project-pages-ssl-certificate-creation-repeatedly-failing/93974/4?u=whitecoop)?
From my machine, when I run
$ dig palousedata.co
I get a valid response, but from two different machines/hosts that the gitlab.com troubleshooter was using, digging palousedata.co gave back SERVFAIL
Reply #3 (from me)
Another way to put it, while I thinking about it: are the errors here (https://check.njal.la/dns/?name=palousedata.co) expected? It’s currently showing nameserver errors
Reply #4 (from Njalla)
There was an issue with DNSSEC for your domain, we have resubmitted the dnssec keys to the registry now and it the domain should be working. We will review what might have caused the DNSSEC failure to make sure it does not happen again in the future.
Oh great, thanks for the transparency. I had a thought about DNSSEC but did not want to go there initially, after seeing the nameserver glue record errors as a first starting point. Context: I know how DNSSEC works, but debugging is hard, explaining it is even harder.
But it makes sense now - DNSSEC ensures the chain of trust of
and I can now access my Gitlab page via HTTPS with my custom domain name.