DNS Resolution Broken? Fast-Track Root Cause Analysis with dig
A systematic methodology for diagnosing DNS resolution failures using dig, covering client-side configuration, recursive and authoritative server queries, DNSSEC validation, negative caching, and Kubernetes DNS troubleshooting with practical command examples.
Establish the Query Chain
Running dig <domain> once proves nothing. Without a server argument, dig uses the system resolver, which may differ from the application's resolver library, search domains, and cache. The correct troubleshooting order: confirm client configuration, query the recursive DNS directly, query the authoritative DNS directly, trace delegation and DNSSEC if needed, then change configuration.
Confirm What the Client Actually Asks
/etc/resolv.confcan be rewritten by systemd-resolved, NetworkManager, DHCP, or container runtimes. Inspecting the file alone is insufficient; resolvectl status shows per-interface DNS, routing domains, and effective configuration.
# Code 04: Confirm resolv.conf source and resolved effective config
readlink -f /etc/resolv.conf
sed -n '1,120p' /etc/resolv.conf
resolvectl status
resolvectl dns
resolvectl domaingetent ahostsreflects the glibc application view (NSS, search domains, local resolver). If getent succeeds but dig @recursive fails, the client likely used a different DNS, cache, or /etc/hosts; the reverse also holds. Distinguish the path.
# Code 05: Query from system resolver perspective, check hosts priority
getent ahosts '<domain>'
grep -E '^[[:space:]]*hosts:' /etc/nsswitch.conf
grep -n -- '<domain>' /etc/hosts || true# Code 06: Query from systemd-resolved view and fetch logs
resolvectl query '<domain>'
systemctl status systemd-resolved --no-pager
journalctl -u systemd-resolved --since '30 minutes ago' --no-pager | tail -n 150If only a specific VPN, interface, or office network fails, check split-DNS routing domains first; do not globally replace resolv.conf.
Query the Recursive DNS Directly
Adding @<recursiveDNS_IP> sends the query straight to that recursive server, bypassing the system resolver. Check A, AAAA, CNAME, and any record types the service needs; querying only A misses IPv6 or CNAME chain issues.
# Code 07: Check common records on the same recursive DNS
RECURSOR='<recursiveDNS_IP>'
dig @"${RECURSOR}" '<domain>' A +noall +answer +comments +stats
dig @"${RECURSOR}" '<domain>' AAAA +noall +answer +comments +stats
dig @"${RECURSOR}" '<domain>' CNAME +noall +answer +comments +stats
dig @"${RECURSOR}" '<domain>' SOA +noall +answer +authority +commentsTTL is a caching semantic, not a permanent validity guarantee. A recursive response TTL lower than the authoritative TTL usually means the cache has partially expired; TTL 0 does not guarantee all applications immediately see the new value. You must query the authoritative server directly to compare.
# Code 08: View recursive response flags, authority, and stats
dig @'<recursiveDNS_IP>' '<domain>' A +multiline +comments +authority +additional +statsThe aa flag means authoritative answer, ra means recursion available, rd means recursion desired. Enterprise recursive servers commonly show rd ra; public authoritative servers should not offer recursion to external clients. Flags are only part of the evidence; combine with server role.
UDP timeout with TCP success points to UDP/53 path, ACL, MTU, or middlebox policy — not record content. nc UDP success does not prove a DNS response was received and cannot replace dig.
# Code 09: Compare UDP vs TCP for the same query
dig @'<recursiveDNS_IP>' '<domain>' A +notcp +time=2 +tries=1 +stats
dig @'<recursiveDNS_IP>' '<domain>' A +tcp +time=2 +tries=1 +statsLow-frequency repeated queries collect reproducible latency and status evidence; they are not load tests. Overall QPS, CPU, and packet loss should come from actual monitoring.
# Code 10: Low-frequency sampling of recursive DNS responses
#!/usr/bin/env bash
set -euo pipefail
DNS_SERVER='<recursiveDNS_IP>'
NAME='<domain>'
for item in 1 2 3 4 5; do
date -Is
dig @"${DNS_SERVER}" "${NAME}" A +time=2 +tries=1 +noall +comments +stats
sleep 2
doneQuery the Authoritative Server Directly
First fetch NS and SOA to identify delegation targets and zone version, then query each confirmed authoritative IP. Do not declare recovery because one NS answers correctly; primary/secondary sync drift, anycast nodes, or BIND views can yield different answers.
# Code 11: Retrieve NS, glue, and SOA
dig '<domain>' NS +noall +answer +additional +comments
dig '<domain>' SOA +noall +answer +authority +commentsWhen querying authoritative servers directly, always use +norecurse to avoid mistaking recursive results for zone data. For NXDOMAIN, the SOA in the authority section is key evidence for the negative answer.
# Code 12: Direct authoritative data query
dig @'<authoritativeDNS_IP>' '<domain>' A +norecurse +noall +answer +authority +comments
dig @'<authoritativeDNS_IP>' '<internalDomain>' SOA +norecurse +noall +answer +authority +commentsIf only internal networks, VPNs, or certain source subnets receive different IPs, suspect BIND views. You must run the same direct authoritative query from both the failing client network and a working client network; testing only on the DNS host itself is insufficient because its source address may hit a different view.
# Code 13: Same authoritative query from different network locations
dig @'<authoritativeDNS_IP>' '<internalDomain>' A +norecurse +comments +answer +authority
dig @'<authoritativeDNS_IP>' '<internalDomain>' AAAA +norecurse +comments +answer +authorityBIND configuration and zone file changes are high-risk. Back up config and zones, keep source files in version control, run named-checkconf and named-checkzone, then reload a single canary DNS. Confirm primary/secondary, zone transfers, and failure domains across multiple authoritative servers; never restart them simultaneously.
// Code 14: /etc/bind/named.conf.options recursion access boundary example
acl trusted_clients {
10.0.0.0/8;
192.168.0.0/16;
localhost;
};
options {
directory "/var/cache/bind";
recursion yes;
allow-recursion { trusted_clients; };
listen-on { any; };
};The example only illustrates syntax; the sample subnets are not real trusted networks. Public authoritative DNS typically disables recursion; enterprise recursive DNS must restrict allowed clients.
; Code 15: /etc/bind/zones/db.<internalDomain>
$TTL 300
@ IN SOA ns1.<internalDomain>. hostmaster.<internalDomain>. (
2026072901 ; serial
3600 ; refresh
600 ; retry
1209600 ; expire
300 ; negative cache TTL
)
IN NS ns1.<internalDomain>.
ns1 IN A <authoritativeDNS_IP>
app IN A <applicationIPv4>The SOA serial increment rule must match the automated release pipeline. The following script does not modify content; it chains backup, validation, reload, and local authoritative query — stopping on failure so errors do not propagate to other nodes.
# Code 16: Validate and load reviewed zone on a single BIND node
#!/usr/bin/env bash
set -euo pipefail
ZONE='<internalDomain>'
ZONE_FILE="/etc/bind/zones/db.${ZONE}"
BACKUP_DIR='/var/backups/bind'
sudo install -d -m 0700 "${BACKUP_DIR}"
sudo cp -a "${ZONE_FILE}" "${BACKUP_DIR}/$(basename "${ZONE_FILE}").$(date +%F-%H%M%S)"
sudo named-checkconf
sudo named-checkzone "${ZONE}" "${ZONE_FILE}"
sudo rndc reload "${ZONE}"
dig @127.0.0.1 "${ZONE}" SOA +norecurse +noall +answer +commentsIf reload reveals an error, roll back by restoring the backed-up zone file, re-run named-checkzone, reload again, then verify SOA serial and resolution. Do not repeatedly systemctl restart bind9 to mask zone content errors.
Trace, Negative Caching, and DNSSEC
dig +tracewalks the public delegation chain, useful for spotting parent-zone NS, glue, child-zone NS, or authoritative node issues. It bypasses the enterprise recursive cache and policy but may fail due to egress restrictions, so it cannot replace querying the enterprise recursive DNS.
# Code 17: Trace public domain delegation chain
dig '<domain>' A +trace +dnssecFor non-existent names, authoritative servers often include an SOA in the authority section. The recursive server caches the NXDOMAIN per negative-caching semantics; if a record was just added but still appears missing, compare authoritative response, recursive response, and SOA before flushing caches globally.
# Code 18: Compare recursive and authoritative negative response evidence
dig @'<authoritativeDNS_IP>' 'missing.<domain>' A +norecurse +noall +authority +comments
dig @'<recursiveDNS_IP>' 'missing.<domain>' A +noall +authority +comments +statsDNSSEC problems often manifest as recursive SERVFAIL while direct authoritative queries return records. Request DNSSEC data, observe the ad flag, RRSIG, DS, and DNSKEY chain; do not permanently disable validation for temporary success.
# Code 19: Check DNSSEC records and validation info
dig @'<recursiveDNS_IP>' '<domain>' A +dnssec +multiline +comments
dig '<domain>' DS +dnssec +noall +answer +authority +comments
dig '<domain>' DNSKEY +dnssec +noall +answer +commentsLarge responses may trigger UDP fragmentation or be dropped by middleboxes. Test with a conservative 1232-byte EDNS buffer size, then compare with TCP. If only large UDP responses fail, check path MTU, DNS inspection, and recursive EDNS policy — do not arbitrarily change records.
# Code 20: Test EDNS UDP size and TCP fallback
dig @'<recursiveDNS_IP>' '<domain>' DNSKEY +dnssec +bufsize=1232 +stats
dig @'<recursiveDNS_IP>' '<domain>' DNSKEY +dnssec +tcp +statsNetwork, Cache, and Container Layer Evidence
When dig times out, the critical divide is whether the DNS server received the request. On the server, capture UDP/TCP 53 briefly; output may contain internal domains and IPs — treat as sensitive troubleshooting data, limit retention and access, never upload to uncontrolled locations.
# Code 21: Short-lived capture of specific client traffic on authoritative DNS node
sudo timeout 30 tcpdump -ni '<interface>' \
'host <failingClientIP> and (udp port 53 or tcp port 53)' \
-vvIf the capture shows requests but no responses, examine named, logs, ACLs, views, and upstreams. If no requests appear, the issue lies in the client-to-server path, wrong DNS address, or middlebox.
# Code 22: Check BIND process, listeners, logs, and config
systemctl status named --no-pager || systemctl status bind9 --no-pager
sudo ss -luntp '( sport = :53 )'
sudo journalctl -u named -n 200 --no-pager || sudo journalctl -u bind9 -n 200 --no-pager
sudo named-checkconfFlushing cache is an impactful operation: subsequent queries go upstream, temporarily increasing upstream load, and it does not fix incorrect authoritative records or wrong DNS addresses. Only flush the minimal name set after confirming authoritative data is correct, recursive cache is proven problematic, and the responsible owner approves. First verify rndc help shows version support.
# Code 23: Flush a single name from recursive cache after approval
sudo rndc help | grep -F flushname
sudo rndc flushname '<domain>'
dig @127.0.0.1 '<domain>' A +noall +answer +comments +statsClient-side flush only helps distinguish local cache from upstream issues; it is not a production fix. Record current state first, then operate on a single failing host.
# Code 24: Flush one Ubuntu client's resolved cache
resolvectl statistics
sudo resolvectl flush-caches
resolvectl query '<domain>'
resolvectl statisticsKubernetes adds Pod resolv.conf, CoreDNS, and possibly NodeLocal DNSCache layers. All kubectl commands below specify the namespace explicitly. Run dig inside the failing Pod and collect CoreDNS logs first; deleting the CoreDNS Pod should not be the first reaction.
# Code 25: Troubleshoot Pod-to-CoreDNS path in a specific namespace
kubectl -n '<namespace>' exec '<podName>' -- cat /etc/resolv.conf
kubectl -n '<namespace>' exec '<podName>' -- dig '<domain>' A +noall +answer +comments +stats
kubectl -n '<namespace>' get pods -l k8s-app=kube-dns -o wide
kubectl -n '<namespace>' logs deployment/coredns --tail=200Prometheus metric names and labels must match what the actual exporter exposes. Without an exporter, do not assume a dns metric exists; correlate SERVFAIL, timeouts, recursive latency, cache hit changes, and process availability with the same time windows used by dig.
# Code 26: Pseudo-code — metric names must match actual exporter output
sum(rate(<dns_servfail_counter>[5m]))
/
sum(rate(<dns_query_counter>[5m]))Evidence Collection and Closure
A DNS incident is not closed because one dig turns green. Preserve: failing name and type, affected networks, default resolver, recursive and authoritative queries, status codes, answers, TTLs, SOA serial, logs or packet captures, change details, and repeated verification. The following script only gathers evidence — it does not flush caches, reload services, or modify DNS.
# Code 27: dns-evidence.sh — collect client, recursive, authoritative evidence
#!/usr/bin/env bash
set -euo pipefail
NAME='<domain>'
RECURSOR='<recursiveDNS_IP>'
AUTH='<authoritativeDNS_IP>'
OUT="dns-evidence-$(date -u +%Y%m%dT%H%M%SZ).txt"
{
echo "timestamp=$(date -Is)"
echo '== client resolver =='
resolvectl status || true
echo '== default query =='
dig "${NAME}" A +noall +answer +authority +comments +stats
echo '== recursive query =='
dig @"${RECURSOR}" "${NAME}" A +noall +answer +authority +comments +stats
echo '== authoritative query =='
dig @"${AUTH}" "${NAME}" A +norecurse +noall +answer +authority +comments
} > "${OUT}"
chmod 0600 "${OUT}"
echo "Written to ${OUT}"When you can answer which client asked which DNS, what each server returned, what the authoritative data is, and at which hop the difference appears, the DNS failure transforms from a vague symptom into a verifiable problem. Only then can you decide whether the fix is correcting a record, repairing delegation, adjusting ACLs, restoring connectivity, flushing a specific cache, or correcting the client DNS — with clear impact scope and rollback justification.
Signed-in readers can open the original source through BestHub's protected redirect.
This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactand we will review it promptly.
How this landed with the community
Was this worth your time?
0 Comments
Thoughtful readers leave field notes, pushback, and hard-won operational detail here.
