Practical Guide to Troubleshooting and Resolving DNS Issues

Overview

DNS (Domain Name System) is a hierarchical, delegated and cached service that resolves a name to an IP address. A lookup traverses multiple layers – application cache, OS cache, /etc/hosts, the local resolver, root servers, TLD servers and finally the authoritative name servers.

Failure categories

Complete resolution failure : NXDOMAIN or SERVFAIL caused by expired domains, down authority servers or network outages.

Incorrect results : Cache poisoning, DNS hijacking or mis‑configured records.

High latency : Overloaded resolvers, long recursive chains, DNSSEC fallback or oversized UDP packets.

Intermittent failures : Partial resolver failures, UDP loss or conntrack table exhaustion (common in Kubernetes).

OS‑level issues : Overwritten /etc/resolv.conf, wrong nsswitch.conf or conflicting systemd-resolved settings.

Core concepts

Record types most relevant to troubleshooting are A, AAAA, CNAME, NS, SOA, PTR, MX, TXT and SRV. TTL controls caching – a short TTL speeds propagation, a long TTL delays it. EDNS0 raises the UDP payload limit from 512 bytes to 4096 bytes; firewalls that block packets larger than 512 bytes can cause silent failures. DNSSEC adds cryptographic validation; a failed validation returns SERVFAIL.

Environment requirements

Ubuntu 24.04 LTS or Rocky Linux 9.5 (LTS releases)

Linux kernel 6.12+ (latest network stack)

systemd 256+ (includes systemd-resolved)

bind‑utils / dnsutils 9.20+ (provides dig, nslookup, nsupdate)

dog 0.1.0+ (modern Rust DNS client)

tcpdump 4.99+ (packet capture)

CoreDNS 1.12+ (Kubernetes default DNS)

BIND9 9.20+ (recursive/authoritative server)

Unbound 1.22+ (high‑performance recursive resolver)

Prometheus 3.x and Grafana 11.x (metrics collection and visualization)

Step‑by‑step troubleshooting

1. Install diagnostic tools

# Ubuntu 24.04
sudo apt update
sudo apt install -y dnsutils mtr-tiny tcpdump whois iputils-ping
# Rocky Linux 9.5
sudo dnf install -y bind-utils mtr tcpdump whois iputils

2. Verify current DNS configuration

# Show resolv.conf
cat /etc/resolv.conf
ls -l /etc/resolv.conf
# Show nsswitch order
grep ^hosts /etc/nsswitch.conf
# systemd‑resolved status
systemctl is-active systemd-resolved
resolvectl status
resolvectl statistics

3. Establish a baseline

Measure normal query latency (cached 0‑5 ms, uncached 20‑100 ms) with a looped dig command.

for i in $(seq 1 10); do
  dig +noall +stats example.com | grep "Query time"
done

4. Deep dive with dig

Basic query: dig example.com Specific type: dig example.com AAAA, dig example.com MX Query a particular server: dig @8.8.8.8 example.com Full resolution trace: dig +trace example.com Compact output: dig +short example.com DNSSEC validation: dig +dnssec example.com Reverse lookup:

dig -x 93.184.216.34

5. Alternative tools

nslookup

and host provide quick checks. tcpdump can capture DNS traffic when higher‑level tools do not reproduce the issue.

# Capture all DNS traffic
sudo tcpdump -i any port 53 -nn -l
# Capture only queries for a name
sudo tcpdump -i any port 53 -nn -l | grep "example.com"

6. Layer‑by‑layer troubleshooting

Local files

# /etc/hosts
cat /etc/hosts
ls -l /etc/hosts

/etc/resolv.conf

# Show content
cat /etc/resolv.conf
# Identify manager (NetworkManager, systemd‑resolved, static)
ls -l /etc/resolv.conf
# Example robust resolv.conf
nameserver 10.0.0.2
nameserver 10.0.0.3
search corp.example.com
options timeout:2 attempts:2 rotate single-request-reopen

systemd‑resolved

# Status and configuration
systemctl status systemd-resolved
resolvectl status
# Flush cache
resolvectl flush-caches
# Enable debug logging
sudo mkdir -p /etc/systemd/resolved.conf.d
cat <<'EOF' | sudo tee /etc/systemd/resolved.conf.d/debug.conf
[Resolve]
LogLevel=debug
EOF
sudo systemctl restart systemd-resolved
journalctl -u systemd-resolved -f --no-pager

BIND9

# Service status
sudo systemctl status named
# Syntax check
sudo named-checkconf /etc/named.conf
sudo named-checkzone example.com /var/named/example.com.zone
# Statistics and cache control
sudo rndc stats
sudo rndc flush
sudo rndc flushname example.com

CoreDNS (Kubernetes)

# Pods and logs
kubectl -n kube-system get pods -l k8s-app=kube-dns
kubectl -n kube-system logs -l k8s-app=kube-dns --tail=100
# ConfigMap
kubectl -n kube-system get configmap coredns -o yaml
# Test from a pod
kubectl run dns-test --image=busybox:1.36 --rm -it --restart=Never -- nslookup example.com

Real‑world cases

Case 1 – resolv.conf overwritten by NetworkManager

# Verify file type
cat /etc/resolv.conf
ls -l /etc/resolv.conf
# Check NetworkManager status
systemctl is-active NetworkManager
# Show DNS mode (default = overwrite)
grep -i dns /etc/NetworkManager/NetworkManager.conf
# Disable automatic handling (option A)
cat <<'EOF' | sudo tee /etc/NetworkManager/conf.d/dns-none.conf
[main]
dns=none
EOF
sudo systemctl restart NetworkManager
# Or set static DNS via nmcli (option B)
sudo nmcli connection modify "eth0" ipv4.dns "10.0.0.2 10.0.0.3"
sudo nmcli connection modify "eth0" ipv4.ignore-auto-dns yes
sudo nmcli connection up "eth0"
# Verify
cat /etc/resolv.conf
dig +short internal-api.corp.example.com

Case 2 – Java startup slowdown due to AAAA timeout

# Add socket‑separation to avoid the conntrack race
echo "options single-request-reopen" | sudo tee -a /etc/resolv.conf
# JVM‑level fix (preferred when IPv6 is not needed)
# Add to Java options: -Djava.net.preferIPv4Stack=true

Case 3 – Split‑DNS conflict between internal and public servers

# systemd‑resolved split‑DNS example
cat <<'EOF' | sudo tee /etc/systemd/resolved.conf.d/split-dns.conf
[Resolve]
DNS=10.0.0.2 10.0.0.3
FallbackDNS=8.8.8.8 1.1.1.1
Domains=corp.example.com
EOF
sudo systemctl restart systemd-resolved
# Verify that internal domains are only sent to the internal DNS
resolvectl query internal-api.corp.example.com

Health‑check script (simplified)

#!/bin/bash
# dns_health_check.sh – monitors a set of domains against multiple resolvers
DNS_SERVERS=("10.0.0.2" "10.0.0.3" "8.8.8.8")
declare -A TEST_DOMAINS=(
  ["internal-api.corp.example.com"]="10.0.1.50"
  ["db-master.corp.example.com"]="10.0.2.10"
  ["example.com"]=""
  ["google.com"]=""
)
LATENCY_THRESHOLD=500   # ms
WEBHOOK_URL="https://example.com/webhook"
# Function definitions omitted for brevity – the script runs dig against each server,
# checks for NXDOMAIN/SERVFAIL, validates expected IP (if any) and compares query time
# against $LATENCY_THRESHOLD, then posts a JSON payload to $WEBHOOK_URL on failure.

Best practices & precautions

Deploy at least two local resolvers in different availability zones; configure rotate, timeout and attempts in resolv.conf.

Use Keepalived or VRRP to provide a virtual IP for DNS HA.

Enable local caching (Unbound or dnsmasq) and pre‑warm the cache after a restart.

Restrict recursion to internal networks (BIND ACLs) and enable response‑rate limiting to mitigate amplification attacks.

Prefer DNS‑over‑TLS for external queries when privacy is required.

In Kubernetes, lower ndots or use fully‑qualified domain names to avoid unnecessary internal retries; consider single-request-reopen or NodeLocal DNSCache to eliminate the conntrack race.

Monitoring & alerting

Key metrics – latency, failure rate, cache hit ratio, QPS, TCP vs UDP ratio – should be scraped with Prometheus. A typical Blackbox Exporter module looks like:

modules:
  dns_internal:
    prober: dns
    timeout: 5s
    dns:
      query_name: "internal-api.corp.example.com"
      query_type: "A"
      valid_rcodes: [NOERROR]
      validate_answer_rrs:
        fail_if_none_matches_regexp: [".*10\\.0\\.1\\.50.*"]
      preferred_ip_protocol: "ip4"

Prometheus scrape config (simplified):

scrape_configs:
  - job_name: 'dns_probe'
    metrics_path: /probe
    params:
      module: [dns_internal]
    static_configs:
      - targets: ['10.0.0.2', '10.0.0.3', '8.8.8.8']
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: blackbox-exporter:9115

Typical alert rules:

- alert: DNSResolutionSlow
  expr: probe_dns_lookup_time_seconds > 0.2
  for: 5m
  labels:
    severity: warning
  annotations:
    summary: "DNS resolution latency > 200 ms"
    description: "Resolver {{ $labels.instance }} is slow ({{ $value }} s)."
- alert: DNSResolutionFailed
  expr: probe_success == 0
  for: 2m
  labels:
    severity: critical
  annotations:
    summary: "DNS resolution failure"
    description: "Resolver {{ $labels.instance }} failed to resolve the test domain."

Backup & recovery

Backup script (simplified) captures all client‑side and server‑side DNS configuration files.

#!/bin/bash
BACKUP_DIR="/backup/dns/$(date +%Y%m%d_%H%M%S)"
mkdir -p "$BACKUP_DIR"
# Client configuration
cp /etc/resolv.conf "$BACKUP_DIR/"
cp /etc/nsswitch.conf "$BACKUP_DIR/"
cp /etc/hosts "$BACKUP_DIR/"
cp -r /etc/systemd/resolved.conf.d "$BACKUP_DIR/" 2>/dev/null
# Server configuration (if present)
[ -d /etc/named ] && cp -r /etc/named "$BACKUP_DIR/bind9"
[ -d /var/named ] && cp -r /var/named "$BACKUP_DIR/bind9-zones"
[ -d /etc/unbound ] && cp -r /etc/unbound "$BACKUP_DIR/unbound"
[ -d /etc/dnsmasq.d ] && cp -r /etc/dnsmasq.d "$BACKUP_DIR/dnsmasq"
# Record current resolution state for later verification
dig +short example.com > "$BACKUP_DIR/resolve_baseline.txt"
resolvectl status > "$BACKUP_DIR/resolved_status.txt" 2>/dev/null
# Archive
tar czf "${BACKUP_DIR}.tar.gz" -C "$(dirname $BACKUP_DIR)" "$(basename $BACKUP_DIR)"
rm -rf "$BACKUP_DIR"
echo "DNS configuration archived to ${BACKUP_DIR}.tar.gz"

Restoration consists of extracting the archive, copying files back to their original locations, restarting the relevant services (e.g., systemd-resolved, named, unbound) and re‑validating with dig or resolvectl query.

Conclusion

The guide walks through the DNS resolution process, categorises five failure modes, explains record types, TTL, EDNS and DNSSEC, and then provides a systematic, layered troubleshooting methodology – from local files to systemd‑resolved, BIND9, Unbound and CoreDNS. Real‑world case studies illustrate common pitfalls such as overwritten resolv.conf, IPv6‑related timeouts and split‑DNS conflicts. The accompanying health‑check script, Prometheus monitoring configuration and backup procedures round out a production‑ready DNS operations playbook.