DNS Troubleshooting - Assumptions and Problem Breakdown

DNS Troubleshooting - Assumptions and Problem Breakdown

1. Troubleshooting
-Assumptions and Problem Breakdown -
Matsuzaki ʻmazʼ Yoshinobu
<[email protected]>
bdNOG18 [email protected] 1

2. Simple tools are useful
•ping, traceroute, dig, and etc.
•ping:
•More powerful if you know
•The assumptions
•How to read the result
bdNOG18 [email protected] 2

3. dig soabd @dns.bd
1.Name resolution of ”dns.bd” with the hostʼs resolver
•Actually querying ”A” and/or “AAAA” of “dns.bd”
•If name resolution fails, dig ends in error
2.Send “bd SOA” query to the resolved IP addresses
•RD (recursion desired) on by default
bdNOG18 [email protected] 3

4. dig soabd @dns.bd
dns.bd
.bd ccTLD servers
root serversFull-service Resolver
1) resolving “dns.bd”
2) “bd SOA” query
bdNOG18 [email protected] 4

5. when 1) fails, the command fails
dns.bd
.bd ccTLD servers
root serversFull-service Resolver
1) resolving “dns.bd”
bdNOG18 [email protected] 5

6. Possible reasons of the failure
1.Full resolver side
•Service issue (IP reachability, packet filtering)
•named issue (process, capability, configuration)
2.Client side
•Reachability issue (IP reachability, packet filtering)
•No resolver (local resolver, nameserver configuration)
3.Authoritative server side
•Service issue (IP reachability, packet filtering)
•named issue (process, capability, configuration)
•zone configuration issue (zone cut, DNSSEC, transfer)
Engineerscan point outthe specific reason
bdNOG18 [email protected] 6

7. Need some !যাগাড়in case of DNS issue
•Cannot use hostname
•No problem to include QNAME in query though
•Cannot rely on Full-service Resolver functionality
•Cache contents
•Recursive mode
•DNSSEC validation
bdNOG18 [email protected] 7

8. Where to start
1.Try another Full-service resolver
•Open DNS services
•Ex. $ dig soabd @1.1.1.1
2.Ensure you have a healthy Internet connection
•Especially TCP/53 and UDP/53 for DNS troubleshooting
3.Isolating the problem by querying authoritative servers
•IPv6 and IPv4 are different protocol
•The response can vary depending on how zone information is
cofingured
bdNOG18 [email protected] 8

9. Querying one by one
•$ dig +norecNS bd @a.root-servers.net
•+norec: To disable recursion, off the RD (Recursive Desired) bit
•NS bd : QTYPE “NS” and QNAME “bd”
•@a.root-servers.net: sending the query to a.root-servers.net
# Assuming a.root-servers.netis resolvable
•Expecting Glue records
•NS records for bd
•A and AAAA records for the bd NS servers
bdNOG18 [email protected] 9

10. 4 NS hosts, 8 IP addresses
•4 hosts serving as bd ccTLD nameservers
•dns.bd, jamuna.btcl.net.bd, surma.btcl.net.bd, and
bd-ns.anycast.pch.net
•Each host has IPv6 and IPv4 addresses
•Send a direct query to the individual IP addresses
•8 times of ”dig +norecSOA bd @<IP address>”
bdNOG18 [email protected] 10

11. Observations at ”that” time
name-serversQuery: SOA bd
dns.bd
jamuna.btcl.net.bd
surma.btcl.net.bd
SERVFAIL
bd-ns.anycast.pch.netSOA serial 2023060867
; <<>> DiG9.10.6 <<>> ns bd @1.1.1.1
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<-opcode: QUERY, status: NOERROR, id: 41885
;; flags: qrrdra; QUERY: 1, ANSWER: 4, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
; OPT=15: 00 09 6e 6f 20 53 45 50 20 6d 61 74 63 68 69 6e 67 20 74 68 65 20 44 5
3 20 66 6f 75 6e 64 20 66 6f 72 20 62 64 2e
("..no SEP matching the DS found for bd.")
; OPT=15: 00 17 31 32 33 2e 34 39 2e 31 32 2e 31 31 32 3a 35 33 20 72 63 6f 64 6
5 3d 53 45 52 56 46 41 49 4c 20 66 6f 72 20 62 64 20 44 4e 53 4b 45 59
("..123.49.12.112:53 rcode=SERVFAIL for bd DNSKEY")
Only one out of four could
responds without DNS error
DNSSEC failure on bd
bdNOG18 [email protected] 11

12. Some tricky parts
*1) gov.bd, com.bd, net.bd, org.bd, ac.bd, and so on
•Those are not DNSSEC-signed
•If you did “dig [email protected]”
at ”that” time, it worked as expected
name-serversSOA bdSOA for subdomains (*1)
dns.bd
jamuna.btcl.net.bd
surma.btcl.net.bd
SERVFAILOK
bd-ns.anycast.pch.netSOA serial 2023060867OK
bdNOG18 [email protected] 12

13. Several zones in a server
•Even the parent (bd) zone is failed,
the servers can reply an answer from
its subdomain (ex. net.bd) zone
.bd ccTLD serverswww.bdren.net.bdquery
bd zonegov.bdzone
com.bdzone
net.bdzone
bdNOG18 [email protected] 13

14. My guess at ”that” time
•3 nameservers failed to load bd zone file
•Could be some DNSSEC singing issues, as other un-signed
subdomains were loaded as expected
•1 nameserver kept (old) bd zone file and answering
•RRSIG (digital signature by DNSSEC) was expired
•This caused DNSSEC verification error
•Worked as expected for the system, but unexpected for users
bdNOG18 [email protected] 14

15. Some possible improvements
•Monitoring
•Zone filegeneration
•Singing and transfer
•SOA serial syncamong nameservers
•Point of Contact
•Troubleinformation
•Technical information
bdNOG18 [email protected] 15

16. Example: JP DNS
•The nameservers for the .jpccTLD are managed by the
JPRS, the .jpregistry, with the cooperation of various
organizations in the Japanese internet industry.
Server nameOrganization
a.dns.jpJPRS (.jpRegistry)
b.dns.jpJPNIC (Japan NIR)
c.dns.jpJPRS
d.dns.jpIIJ (Commercial ISP)
e.dns.jpWIDE (Research Consortium)
f.dns.jpNII (Academic Research Institute)
g.dns.jpJPRS
h.dns.jpJPRS
bdNOG18 [email protected] 16