Skip to content
DNS Resolution Broken? Fast-Track Root Cause Analysis with dig
Operations 23 min read

DNS Resolution Broken? Fast-Track Root Cause Analysis with dig

A systematic methodology for diagnosing DNS resolution failures using dig, covering client-side configuration, recursive and authoritative server queries, DNSSEC validation, negative caching, and Kubernetes DNS troubleshooting with practical command examples.

Raymond Ops
Raymond Ops
Raymond Ops

Establish the Query Chain

Running dig <domain> once proves nothing. Without a server argument, dig uses the system resolver, which may differ from the application's resolver library, search domains, and cache. The correct troubleshooting order: confirm client configuration, query the recursive DNS directly, query the authoritative DNS directly, trace delegation and DNSSEC if needed, then change configuration.

Confirm What the Client Actually Asks

/etc/resolv.conf

can be rewritten by systemd-resolved, NetworkManager, DHCP, or container runtimes. Inspecting the file alone is insufficient; resolvectl status shows per-interface DNS, routing domains, and effective configuration.

# Code 04: Confirm resolv.conf source and resolved effective config
readlink -f /etc/resolv.conf
sed -n '1,120p' /etc/resolv.conf
resolvectl status
resolvectl dns
resolvectl domain
getent ahosts

reflects the glibc application view (NSS, search domains, local resolver). If getent succeeds but dig @recursive fails, the client likely used a different DNS, cache, or /etc/hosts; the reverse also holds. Distinguish the path.

# Code 05: Query from system resolver perspective, check hosts priority
getent ahosts '<domain>'
grep -E '^[[:space:]]*hosts:' /etc/nsswitch.conf
grep -n -- '<domain>' /etc/hosts || true
# Code 06: Query from systemd-resolved view and fetch logs
resolvectl query '<domain>'
systemctl status systemd-resolved --no-pager
journalctl -u systemd-resolved --since '30 minutes ago' --no-pager | tail -n 150

If only a specific VPN, interface, or office network fails, check split-DNS routing domains first; do not globally replace resolv.conf.

Query the Recursive DNS Directly

Adding @<recursiveDNS_IP> sends the query straight to that recursive server, bypassing the system resolver. Check A, AAAA, CNAME, and any record types the service needs; querying only A misses IPv6 or CNAME chain issues.

# Code 07: Check common records on the same recursive DNS
RECURSOR='<recursiveDNS_IP>'
dig @"${RECURSOR}" '<domain>' A +noall +answer +comments +stats
dig @"${RECURSOR}" '<domain>' AAAA +noall +answer +comments +stats
dig @"${RECURSOR}" '<domain>' CNAME +noall +answer +comments +stats
dig @"${RECURSOR}" '<domain>' SOA +noall +answer +authority +comments

TTL is a caching semantic, not a permanent validity guarantee. A recursive response TTL lower than the authoritative TTL usually means the cache has partially expired; TTL 0 does not guarantee all applications immediately see the new value. You must query the authoritative server directly to compare.

# Code 08: View recursive response flags, authority, and stats
dig @'<recursiveDNS_IP>' '<domain>' A +multiline +comments +authority +additional +stats

The aa flag means authoritative answer, ra means recursion available, rd means recursion desired. Enterprise recursive servers commonly show rd ra; public authoritative servers should not offer recursion to external clients. Flags are only part of the evidence; combine with server role.

UDP timeout with TCP success points to UDP/53 path, ACL, MTU, or middlebox policy — not record content. nc UDP success does not prove a DNS response was received and cannot replace dig.

# Code 09: Compare UDP vs TCP for the same query
dig @'<recursiveDNS_IP>' '<domain>' A +notcp +time=2 +tries=1 +stats
dig @'<recursiveDNS_IP>' '<domain>' A +tcp +time=2 +tries=1 +stats

Low-frequency repeated queries collect reproducible latency and status evidence; they are not load tests. Overall QPS, CPU, and packet loss should come from actual monitoring.

# Code 10: Low-frequency sampling of recursive DNS responses
#!/usr/bin/env bash
set -euo pipefail
DNS_SERVER='<recursiveDNS_IP>'
NAME='<domain>'
for item in 1 2 3 4 5; do
  date -Is
  dig @"${DNS_SERVER}" "${NAME}" A +time=2 +tries=1 +noall +comments +stats
  sleep 2
done

Query the Authoritative Server Directly

First fetch NS and SOA to identify delegation targets and zone version, then query each confirmed authoritative IP. Do not declare recovery because one NS answers correctly; primary/secondary sync drift, anycast nodes, or BIND views can yield different answers.

# Code 11: Retrieve NS, glue, and SOA
dig '<domain>' NS +noall +answer +additional +comments
dig '<domain>' SOA +noall +answer +authority +comments

When querying authoritative servers directly, always use +norecurse to avoid mistaking recursive results for zone data. For NXDOMAIN, the SOA in the authority section is key evidence for the negative answer.

# Code 12: Direct authoritative data query
dig @'<authoritativeDNS_IP>' '<domain>' A +norecurse +noall +answer +authority +comments
dig @'<authoritativeDNS_IP>' '<internalDomain>' SOA +norecurse +noall +answer +authority +comments

If only internal networks, VPNs, or certain source subnets receive different IPs, suspect BIND views. You must run the same direct authoritative query from both the failing client network and a working client network; testing only on the DNS host itself is insufficient because its source address may hit a different view.

# Code 13: Same authoritative query from different network locations
dig @'<authoritativeDNS_IP>' '<internalDomain>' A +norecurse +comments +answer +authority
dig @'<authoritativeDNS_IP>' '<internalDomain>' AAAA +norecurse +comments +answer +authority

BIND configuration and zone file changes are high-risk. Back up config and zones, keep source files in version control, run named-checkconf and named-checkzone, then reload a single canary DNS. Confirm primary/secondary, zone transfers, and failure domains across multiple authoritative servers; never restart them simultaneously.

// Code 14: /etc/bind/named.conf.options recursion access boundary example
acl trusted_clients {
    10.0.0.0/8;
    192.168.0.0/16;
    localhost;
};

options {
    directory "/var/cache/bind";
    recursion yes;
    allow-recursion { trusted_clients; };
    listen-on { any; };
};

The example only illustrates syntax; the sample subnets are not real trusted networks. Public authoritative DNS typically disables recursion; enterprise recursive DNS must restrict allowed clients.

; Code 15: /etc/bind/zones/db.<internalDomain>
$TTL 300
@ IN SOA ns1.<internalDomain>. hostmaster.<internalDomain>. (
2026072901 ; serial
3600       ; refresh
600        ; retry
1209600    ; expire
300        ; negative cache TTL
)
IN NS ns1.<internalDomain>.
ns1 IN A <authoritativeDNS_IP>
app IN A <applicationIPv4>

The SOA serial increment rule must match the automated release pipeline. The following script does not modify content; it chains backup, validation, reload, and local authoritative query — stopping on failure so errors do not propagate to other nodes.

# Code 16: Validate and load reviewed zone on a single BIND node
#!/usr/bin/env bash
set -euo pipefail
ZONE='<internalDomain>'
ZONE_FILE="/etc/bind/zones/db.${ZONE}"
BACKUP_DIR='/var/backups/bind'

sudo install -d -m 0700 "${BACKUP_DIR}"
sudo cp -a "${ZONE_FILE}" "${BACKUP_DIR}/$(basename "${ZONE_FILE}").$(date +%F-%H%M%S)"
sudo named-checkconf
sudo named-checkzone "${ZONE}" "${ZONE_FILE}"
sudo rndc reload "${ZONE}"
dig @127.0.0.1 "${ZONE}" SOA +norecurse +noall +answer +comments

If reload reveals an error, roll back by restoring the backed-up zone file, re-run named-checkzone, reload again, then verify SOA serial and resolution. Do not repeatedly systemctl restart bind9 to mask zone content errors.

Trace, Negative Caching, and DNSSEC

dig +trace

walks the public delegation chain, useful for spotting parent-zone NS, glue, child-zone NS, or authoritative node issues. It bypasses the enterprise recursive cache and policy but may fail due to egress restrictions, so it cannot replace querying the enterprise recursive DNS.

# Code 17: Trace public domain delegation chain
dig '<domain>' A +trace +dnssec

For non-existent names, authoritative servers often include an SOA in the authority section. The recursive server caches the NXDOMAIN per negative-caching semantics; if a record was just added but still appears missing, compare authoritative response, recursive response, and SOA before flushing caches globally.

# Code 18: Compare recursive and authoritative negative response evidence
dig @'<authoritativeDNS_IP>' 'missing.<domain>' A +norecurse +noall +authority +comments
dig @'<recursiveDNS_IP>' 'missing.<domain>' A +noall +authority +comments +stats

DNSSEC problems often manifest as recursive SERVFAIL while direct authoritative queries return records. Request DNSSEC data, observe the ad flag, RRSIG, DS, and DNSKEY chain; do not permanently disable validation for temporary success.

# Code 19: Check DNSSEC records and validation info
dig @'<recursiveDNS_IP>' '<domain>' A +dnssec +multiline +comments
dig '<domain>' DS +dnssec +noall +answer +authority +comments
dig '<domain>' DNSKEY +dnssec +noall +answer +comments

Large responses may trigger UDP fragmentation or be dropped by middleboxes. Test with a conservative 1232-byte EDNS buffer size, then compare with TCP. If only large UDP responses fail, check path MTU, DNS inspection, and recursive EDNS policy — do not arbitrarily change records.

# Code 20: Test EDNS UDP size and TCP fallback
dig @'<recursiveDNS_IP>' '<domain>' DNSKEY +dnssec +bufsize=1232 +stats
dig @'<recursiveDNS_IP>' '<domain>' DNSKEY +dnssec +tcp +stats

Network, Cache, and Container Layer Evidence

When dig times out, the critical divide is whether the DNS server received the request. On the server, capture UDP/TCP 53 briefly; output may contain internal domains and IPs — treat as sensitive troubleshooting data, limit retention and access, never upload to uncontrolled locations.

# Code 21: Short-lived capture of specific client traffic on authoritative DNS node
sudo timeout 30 tcpdump -ni '<interface>' \
  'host <failingClientIP> and (udp port 53 or tcp port 53)' \
  -vv

If the capture shows requests but no responses, examine named, logs, ACLs, views, and upstreams. If no requests appear, the issue lies in the client-to-server path, wrong DNS address, or middlebox.

# Code 22: Check BIND process, listeners, logs, and config
systemctl status named --no-pager || systemctl status bind9 --no-pager
sudo ss -luntp '( sport = :53 )'
sudo journalctl -u named -n 200 --no-pager || sudo journalctl -u bind9 -n 200 --no-pager
sudo named-checkconf

Flushing cache is an impactful operation: subsequent queries go upstream, temporarily increasing upstream load, and it does not fix incorrect authoritative records or wrong DNS addresses. Only flush the minimal name set after confirming authoritative data is correct, recursive cache is proven problematic, and the responsible owner approves. First verify rndc help shows version support.

# Code 23: Flush a single name from recursive cache after approval
sudo rndc help | grep -F flushname
sudo rndc flushname '<domain>'
dig @127.0.0.1 '<domain>' A +noall +answer +comments +stats

Client-side flush only helps distinguish local cache from upstream issues; it is not a production fix. Record current state first, then operate on a single failing host.

# Code 24: Flush one Ubuntu client's resolved cache
resolvectl statistics
sudo resolvectl flush-caches
resolvectl query '<domain>'
resolvectl statistics

Kubernetes adds Pod resolv.conf, CoreDNS, and possibly NodeLocal DNSCache layers. All kubectl commands below specify the namespace explicitly. Run dig inside the failing Pod and collect CoreDNS logs first; deleting the CoreDNS Pod should not be the first reaction.

# Code 25: Troubleshoot Pod-to-CoreDNS path in a specific namespace
kubectl -n '<namespace>' exec '<podName>' -- cat /etc/resolv.conf
kubectl -n '<namespace>' exec '<podName>' -- dig '<domain>' A +noall +answer +comments +stats
kubectl -n '<namespace>' get pods -l k8s-app=kube-dns -o wide
kubectl -n '<namespace>' logs deployment/coredns --tail=200

Prometheus metric names and labels must match what the actual exporter exposes. Without an exporter, do not assume a dns metric exists; correlate SERVFAIL, timeouts, recursive latency, cache hit changes, and process availability with the same time windows used by dig.

# Code 26: Pseudo-code — metric names must match actual exporter output
sum(rate(<dns_servfail_counter>[5m]))
/
sum(rate(<dns_query_counter>[5m]))

Evidence Collection and Closure

A DNS incident is not closed because one dig turns green. Preserve: failing name and type, affected networks, default resolver, recursive and authoritative queries, status codes, answers, TTLs, SOA serial, logs or packet captures, change details, and repeated verification. The following script only gathers evidence — it does not flush caches, reload services, or modify DNS.

# Code 27: dns-evidence.sh — collect client, recursive, authoritative evidence
#!/usr/bin/env bash
set -euo pipefail
NAME='<domain>'
RECURSOR='<recursiveDNS_IP>'
AUTH='<authoritativeDNS_IP>'
OUT="dns-evidence-$(date -u +%Y%m%dT%H%M%SZ).txt"

{
  echo "timestamp=$(date -Is)"
  echo '== client resolver =='
  resolvectl status || true
  echo '== default query =='
  dig "${NAME}" A +noall +answer +authority +comments +stats
  echo '== recursive query =='
  dig @"${RECURSOR}" "${NAME}" A +noall +answer +authority +comments +stats
  echo '== authoritative query =='
  dig @"${AUTH}" "${NAME}" A +norecurse +noall +answer +authority +comments
} > "${OUT}"

chmod 0600 "${OUT}"
echo "Written to ${OUT}"

When you can answer which client asked which DNS, what each server returned, what the authoritative data is, and at which hop the difference appears, the DNS failure transforms from a vague symptom into a verifiable problem. Only then can you decide whether the fix is correcting a record, repairing delegation, adjusting ACLs, restoring connectivity, flushing a specific cache, or correcting the client DNS — with clear impact scope and rollback justification.

Original Source

Signed-in readers can open the original source through BestHub's protected redirect.

Sign in to view source
Republication Notice

This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactadmin@besthub.devand we will review it promptly.

KubernetesTroubleshootingDNSBINDCoreDNStcpdumpDNSSECdigEDNSrecursive DNSsystemd-resolvedauthoritative DNSnegative cachingrndc
Reader feedback

How this landed with the community

Sign in to like

Rate this article

Was this worth your time?

Sign in to rate
Discussion

0 Comments

Thoughtful readers leave field notes, pushback, and hard-won operational detail here.