How a DNS Resolution Failure Prompted a Full Network Layer Investigation

Incident Overview

A service calling a third‑party API experienced occasional timeouts: success rate dropped from >99.9% to 95%, response time rose from 200 ms to 5‑10 s, and logs filled with UnknownHostException. The root symptom was DNS resolution failure.

Investigation Process

Step 1: Verify DNS Configuration

# 1. View current DNS settings
cat /etc/resolv.conf
# nameserver 114.114.114.114
# nameserver 8.8.8.8

# 2. Manually resolve the domain
nslookup api.thirdparty.com

# 3. Test with different DNS servers
dig @8.8.8.8 api.thirdparty.com

Result: the 114.114.114.114 server sometimes failed, while 8.8.8.8 was stable.

Step 2: Check JVM DNS Cache

// JVM caches DNS results by default
// Success cache TTL: 30 s, failure cache TTL: 10 s
// To view cache, add logging in Java code

Step 3: Locate the Problem

# 1. Inspect /etc/hosts
cat /etc/hosts
# No entry for api.thirdparty.com

# 2. Measure DNS lookup time
time nslookup api.thirdparty.com
# Timeout after 5 s

# 3. Ping the DNS server
ping 114.114.114.114
# Packet loss is intermittent

Root cause: the public DNS server 114.114.114.114 is unstable, causing occasional resolution failures.

Solutions

Solution 1: Switch to Stable DNS Servers

# Edit /etc/resolv.conf
nameserver 8.8.8.8
nameserver 223.5.5.5   # Alibaba DNS
nameserver 119.29.29.29 # Tencent DNS

Solution 2: Configure JVM DNS Cache

# JVM options to adjust DNS cache TTLs
-Dnetworkaddress.cache.ttl=60          # cache successes for 60 s
-Dnetworkaddress.cache.negative.ttl=10 # cache failures for 10 s
-Dsun.net.inetaddr.ttl=60

# Or set in java.security
networkaddress.cache.ttl=60
networkaddress.cache.negative.ttl=10

Solution 3: Add Local Hosts Fallback

# Append a static entry
echo "1.2.3.4 api.thirdparty.com" >> /etc/hosts

Solution 4: Implement Application‑Level Retries

public String callThirdParty() {
    int retryCount = 0;
    while (retryCount < 3) {
        try {
            return restTemplate.getForObject(url, String.class);
        } catch (UnknownHostException e) {
            retryCount++;
            log.warn("DNS resolution failed, retry {}/3", retryCount);
            if (retryCount >= 3) {
                throw new RuntimeException("Third‑party API unavailable", e);
            }
            try { Thread.sleep(1000); } catch (InterruptedException ie) { Thread.currentThread().interrupt(); }
        }
    }
    return null;
}

DNS Troubleshooting Command Cheat Sheet

nslookup <domain>

– query DNS resolution dig <domain> – detailed DNS query dig @8.8.8.8 <domain> – query using a specific DNS server cat /etc/resolv.conf – view DNS configuration time nslookup <domain> – measure lookup latency ipconfig /flushdns – clear DNS cache on Windows systemctl restart systemd-resolved – restart DNS service on Linux

Final Checklist

Are production DNS servers stable?

Is the JVM DNS cache configured appropriately?

Do critical external dependencies have hosts fallback entries?

Is there monitoring and alerting for DNS resolution issues?

If you rely on external APIs, verify your DNS configuration promptly.