DNS Resolution Failures: Diagnose, Recover, and Prevent Service Outages

Real-World Incident: DNS Resolution Failure at 2:17 AM

At 2:17 AM, order service error rate jumped from 0.2% to 38%. Alerts showed dial tcp: lookup mysql-order.internal: i/o timeout, no such host, and Temporary failure in name resolution. Initial assumptions were MySQL failure or network outage, but database was healthy and ping to gateway and 8.8.8.8 succeeded. However, ping mysql-order.internal returned Name or service not known. The root cause: internal CoreDNS lost one replica due to node memory pressure; the remaining replica suffered 60% UDP packet loss under peak QPS. With resolv.conf configured with two nameservers and timeout:5 attempts:2, glibc retry logic waited up to 10 seconds before failing, while business HTTP timeout was only 3 seconds. The symptom was timeout, but the essence was resolution latency exceeding upstream tolerance.

Why DNS Failures Are Hard to Diagnose

Not binary up/down; often slow, partial, intermittent, or domain-specific failures.

Symptoms masquerade as connection timeouts, pool exhaustion, certificate verification failures, HTTP 502/504.

Multiple resolution paths across application, OS, container, and cluster layers (glibc, musl, Go netgo/cgo, JVM cache, Node.js libuv) behave differently.

Caching with TTL means apparent recovery may just be cache expiry; root cause persists.

Applicable Scenarios and Environment Assumptions

The article covers six scenarios: single-host resolution failure, single-host partial domain failure, cluster-wide batch failure, intermittent slowness, container resolution failure, and wrong resolution results. Tested environments include systemd-resolved 253/255/257, dnsmasq 2.90, CoreDNS 1.12.x, glibc 2.34/2.39, musl 1.2.5, Go 1.22+, bind-utils 9.16/9.18, on RHEL/Rocky 9.4-9.6 and Ubuntu 22.04/24.04. Critical note: different distributions manage DNS clients differently (RHEL uses NetworkManager + traditional resolv.conf; Ubuntu uses systemd-resolved with stub at 127.0.0.53; Kubernetes pods get ClusterIP nameserver via kubelet, bypassing host resolved/dnsmasq unless hostNetwork or NodeLocal DNSCache).

30-Second Pre-Check: Is It DNS?

Compare IP vs domain access: curl --max-time 3 http://IP/healthz vs curl --max-time 3 http://domain/healthz. IP works but domain fails → DNS suspected.

Measure name lookup time:

curl -w 'namelookup=%{time_namelookup} connect=%{time_connect} total=%{time_total}
'

. Large namelookup indicates DNS delay.

Direct query with dig: dig +time=2 +tries=1 domain A. If dig works but curl fails, issue is client config/cache (e.g., different resolv.conf, nscd, systemd-resolved cache).

Interpret error codes: NXDOMAIN/no such host = domain missing; SERVFAIL = server failure; REFUSED = ACL/recursion denied; i/o timeout/Temporary failure = no response (UDP loss, firewall, conntrack full); 0.0.0.0 = hijacking; public IP for internal domain = search domain misrouting.

Core Knowledge: Resolution Path

A getaddrinfo("svc.internal") call traverses: application → glibc reads /etc/nsswitch.conf hosts line → files (/etc/hosts) → dns (reads /etc/resolv.conf) → if nameserver is 127.0.0.53, systemd-resolved stub (cache + split) → else direct UDP/TCP 53 to that address → mdns/myhostname → return error codes.

nsswitch.conf: Order of Lookup

Typical: hosts: files dns myhostname (RHEL) or files mdns4_minimal [NOTFOUND=return] dns (Ubuntu). Pitfalls: myhostname can resolve hostname to local IP causing conflicts; [NOTFOUND=return] after mDNS stops further DNS queries, breaking single-label internal names.

resolv.conf: Five Parameters Control Latency

nameserver

(max 3, glibc MAXNS limit) search domains (each adds a retry round on failure) ndots:n (default 1; dots < n triggers search domain attempts first) timeout:n (default 5s per query) attempts:n (default 2 retries)

Worst-case wait = timeout × attempts × nameservers × search domains. Example: timeout:5 attempts:2, 2 nameservers, 2 search domains, querying svc.internal (1 dot < ndots:5) yields up to 20 queries × 5s = 100s theoretical, capped at 30s by glibc. Production recommendation: timeout:2 attempts:2 ndots:2 single-request-reopen with single search domain.

systemd-resolved (Ubuntu Default)

Applications query 127.0.0.53:53. Key commands: resolvectl status (shows current DNS server, DNSSEC mode, resolv.conf mode), resolvectl statistics (cache hits, timeouts), resolvectl flush-caches, resolvectl query --cache=no domain. Configuration via /etc/systemd/resolved.conf (DNS, Domains, DNSSEC, Cache, StaleRetentionSec) requires reload; DNSStubListener changes require restart. Negative caching often causes post-fix failures; StaleRetentionSec (default 0) can serve stale data during upstream outage if set (e.g., 86400) but risks returning decommissioned IPs.

Container Resolution Path

Pod /etc/resolv.conf typically:

search default.svc.cluster.local svc.cluster.local cluster.local

, nameserver 10.96.0.10, options ndots:5. ndots:5 causes short names like mysql-order to try 4 search suffixes before original; use FQDN with trailing dot ( mysql-order.default.svc.cluster.local.) for single query. Nameserver is ClusterIP load-balanced to CoreDNS pods; failure points include conntrack table full, UDP session timeout, CoreDNS replica shortage, routing to ClusterIP.

Five-Layer Troubleshooting Framework

L1 Application: what name, what library, own cache?

L2 Config: what resolv.conf/nsswitch.conf does the process actually read?

L3 Client library: how glibc/musl/Go handle query (search, ndots, retries)

L4 Local resolver: resolved/dnsmasq/CoreDNS health, cache pollution

L5 Upstream/authoritative: forward chain, authoritative zone, DNSSEC, firewall

Most DNS issues are L1/L2. Proceed deeper only when lower layers confirmed correct.

Practical On-Site Steps

Step 1: Identify Failing Names and Frequency

Extract failing hostnames from logs, count occurrences. If names show duplicated suffixes ( svc.internal.svc.internal), it's search domain misconfiguration. Run 20 getent hosts probes to measure success rate and latency patterns.

Step 2: Check Process-Actual Config

Use /proc/<pid>/root/etc/resolv.conf and nsswitch.conf to see what the process actually reads (critical for containers). Verify symlink target exists; broken symlink is a common failure.

Step 3: Bypass Cache, Query Each Level

Run four dig queries: local resolver (A), upstream1 (B), upstream2 (C), iterative trace (D, public only). Compare results using decision table to isolate faulty layer.

Step 4: strace/tcpdump Client Library Behavior

strace -f -e trace=network -s 200 -p <pid>

reveals actual queries sent. Look for duplicated suffixes (search issue), repeated sendto without recvfrom (UDP loss), long sendto loops (timeout too high), ECONNREFUSED to 127.0.0.53 (resolved not listening). Safer: tcpdump -i any udp port 53 with tshark decoding.

Step 5: Verify Local Resolver Health

For systemd-resolved: resolvectl statistics (watch Timeouts, Cache Hits ratio, Current Transactions). For dnsmasq: send SIGUSR1, check logs for forwarded vs local queries. For CoreDNS: check pod status, logs, Prometheus metrics ( coredns_forward_healthcheck_failures_total, rcode=SERVFAIL, cache hit rate).

Step 6: Kernel/Network Evidence

Check conntrack table ( /proc/sys/net/netfilter/nf_conntrack_count vs max, dmesg for table full). UDP receive buffer errors ( nstat -az | grep UdpRcvbufErrors). Path quality with mtr -u -P 53 -c 100 <dns_ip>. Firewall drops via nft list ruleset or iptables -L -v packet counters.

Common Root Causes and Fixes

Search domain/ndots explosion: Reduce search domains, set ndots:2, use FQDN with trailing dot.

UDP packet loss: Fix network, increase CoreDNS replicas, enable single-request-reopen.

conntrack table full: Increase nf_conntrack_max, reduce UDP timeout.

Resolver cache pollution: Flush caches ( resolvectl flush-caches, dnsmasq SIGHUP, CoreDNS rollout).

Negative caching: Stale NXDOMAIN cached; flush or wait TTL.

DNSSEC validation failure: Ensure DNSSEC=allow-downgrade or disable if upstream unsupported.

Resolver process OOM/killed: Set memory limits, monitor.

Configuration drift: Multiple DNS managers (resolved, dnsmasq, nscd) conflict; enforce single manager.

Runtime-Specific Behaviors

Different runtimes resolve differently: glibc (full nsswitch, search, ndots, no cache unless nscd); Go default pure Go resolver (reads resolv.conf, honors ndots/search, no cache, ignores nsswitch, consistent across machines); Go with CGO (uses glibc); JVM (caches positive results forever by default, networkaddress.cache.ttl=-1); Node.js (libuv thread pool default 4, DNS slow blocks pool); Python (blocking getaddrinfo, no timeout param, relies on system timeout); musl (simplified, limited search domain support). Benchmark under 60% UDP loss: dig (2s fail), getent (20s), Go (10s), Python (20s), Node (20s). Dig does not reflect application behavior.

Emergency Runbooks by Service Type

Online API (latency-sensitive): Lower resolv.conf timeout to 1s, reduce connection pool max_wait, ensure local caching, don't restart (causes reconnect storm).

Batch/offline jobs: Pause scheduling, temporarily hardcode IPs in /etc/hosts with expiry comments, fix DNS, resume gradually.

Database connections: Existing connections unaffected; monitor new connection failures via ss -tnp; fix DNS, pre-warm pool.

Preventive Design

Unify resolution path: One DNS client per host (resolved or dnsmasq), single writer for resolv.conf, audit script comparing expected vs actual config.

HA at every layer: ≥2 nameservers from different AZs, CoreDNS replicas ≥3 with topologySpreadConstraints, NodeLocal DNSCache, multiple upstreams with health checks, authoritative NS ≥2.

Application-side caching + timeout control: Cache results 30-60s (TTL ≤ DNS TTL), per-call timeout 1-2s (Go via net.Resolver with custom Dialer).

Monitoring three dimensions: Availability (client-side success rate), Performance (P99 resolve latency), Correctness (probe record matches expected IP).

Chaos drills: Stop one DNS, inject 50% loss with tc, drop all port 53 in staging; measure failover time and degradation.

Recovery Actions by Risk Level

Low: Adjust client params (timeout, attempts) via proper config management (not sed on symlink).

Low: Flush caches (resolvectl, dnsmasq SIGHUP, CoreDNS rollout with pre-scale).

Medium: Runtime DNS switch via resolvectl dns (reverts on restart).

Medium-high: Restart resolver (brief ECONNREFUSED, cache flush spikes upstream).

High: Modify authoritative zone (backup, named-checkzone, rndc reload; TTL delays propagation).

Always have rollback plan recorded; verify fix for 30+ minutes covering peak.

Verification Checklist

dig local resolver: NOERROR, <10ms, 10/10 success

dig each upstream: success, consistent results

getent hosts: rc=0, correct IP

App logs: no new resolve errors for 5 min

Error rate back to baseline

Resolve P99 <50ms

resolvectl cache hits growing

Connection pools stable

Pod internal resolution works, CoreDNS all Ready

Probe failure counter static

Summary

DNS troubleshooting difficulty lies in deceptive symptoms and fragmented paths. Key habits: check error code first; distinguish fast vs slow failure; trust process-view config ( /proc/<pid>/root/etc/resolv.conf); if dig works but app fails, stop checking DNS server; understand timeout×attempts×nameservers×search multiplication; recover from low to high risk; verify over peak; long-term: converge paths, enforce FQDN with trailing dot, don't overload DNS with service discovery.