Production Network Troubleshooting: From DNS to TCP Connection Analysis

Before Troubleshooting: Converge Symptoms

Before running commands, gather four critical pieces of information that determine which steps can be skipped:

Who accesses whom : Source host, target host (both IP and domain name; domain failures 90% stem from resolution or caching).

All or partial : Are all targets failing, or only specific IPs/domains/sources? "Only A to B fails" differs fundamentally from "global failure."

When it started : Precise to the minute, cross-referenced with change timelines (releases, network tickets, cloud provider maintenance).

Failure mode : Immediate failure ( Connection refused, No route to host, resolution failure) vs. hang/timeout (long silence). Immediate failure means "peer explicitly says no"; hang usually means "packets dropped en route" — opposite directions.

Missing any of these leads to misdiagnosis. For example, without "when it started," you cannot correlate the fault with a specific release or network change.

Section 1: DNS Resolution

Goal: Determine whether domain-to-IP resolution works. Follow this sequence:

Step 1: Local Resolution Verification

# Record full latency and result (example output)
# server:
# 10.20.1.36
#
# Query time: 1 msec
# SERVER: 10.20.1.36#53(10.20.1.36)
dig +time=3 +tries=1 app.internal.example.com

+time=3 +tries=1

limits single-query wait and avoids dig's default retries inflating perceived latency. Judgment: If IP returns with normal latency (internal DNS typically milliseconds), skip DNS section; if resolution fails or takes seconds, continue.

Step 2: Distinguish "Resolution Failure" from "Stale/Expired Cache"

Application reports "domain not found" but dig works. Common cause: Application uses /etc/resolv.conf upstream (same as dig default), but Java ( InetAddress with its own DNS cache, negative cache default 10s, positive cache behavior varies by JDK) or middleware with TTL caching (Varnish forward proxy, Nginx resolver directive) may hold stale IPs — especially after target IP migration (Keepalived VIP switch, cloud SLB change). Use strace on the application process to see actual DNS queries and responses:

# Observe DNS packets and results from the process
strace -e trace=network -p <pid> 2>&1 | grep -iE 'getaddrinfo|resolv|connect'

Step 3: Verify /etc/resolv.conf Configuration

In containers, resolv.conf is injected by K8s kube-dns/CoreDNS or Docker networking; host config does not apply. Enter the container to inspect:

cat /etc/resolv.conf
# Record nameserver, search domains, options

Overlong search domains amplify short-name resolution failure probability (short name tries each search domain sequentially, all time out before FQDN). High options ndots (e.g., ndots:5) causes short names to be attempted as FQDN first before falling back to search domains — a common source of "slow resolution" in internal networks.

Step 4: Validate Upstream DNS Server

Local resolution works but other machines fail, or suspect DNS server overload:

# Directly query suspect upstream (example: cloud internal DNS)
dig @10.20.1.36 app.internal.example.com +time=3 +tries=1

# 20 consecutive queries to check for failures or intermittent timeouts (example stats)
# 20 queries, 0 failed
for i in $(seq 20); do dig +time=1 +tries=1 app.internal.example.com >/dev/null 2>&1 || echo "fail $i"; done

Intermittent failures (1–2 out of 20) point to DNS server overload or path packet loss, not configuration error — global config errors yield 100% failure.

Common DNS root causes: resolv.conf nameserver pointing to decommissioned DNS; search and ndots amplifying timeouts; DNS server overwhelmed by high-frequency queries from connection-pool exhaustion causing many apps to concurrently resolve same domain without caching; long negative cache TTL. Remediation: Backup resolv.conf before changes; for DNS server overload, add local caching ( dnsmasq, nscd, or application-framework cache) rather than blindly adding servers.

Section 2: Layer 3 Routing & Connectivity

After DNS exclusion (or if failure initially reported as IP not domain), verify layer 3 reachability.

Step 1: Local Routing Table & Egress Interface

# See which route the target IP takes (example output)
# 10.30.2.50 via 10.20.1.254 dev eth0
ip route get 10.30.2.50

# Full table (only entries relevant to target subnet needed)
ip route

ip route get

is the core command for "local decision": it shows source IP, destination IP, next hop, and egress interface. If next hop points to a decommissioned switch port, or egress interface isn't the expected one (common in dual-NIC setups), root cause is local routing.

Step 2: Hop-by-Hop Probe to Locate Break Point

# mtr combines traceroute and ping (example output, 30 seconds)
# HOST: web-01          Loss%   Snt   Avg  Last
# 10.20.1.254            0.0%    30   0.8  0.9
# 10.30.1.254            0.0%    30   1.2  1.1
# 10.30.2.50            36.7%    30   8.4  9.1
mtr -n -c 30 10.30.2.50

Judgment: Watch Loss% per hop. First two hops clean, persistent high loss from a certain hop onward (including target) means break is "between that hop and the next". Loss only on last hop with clean intermediate hops suggests target host itself (overload, interface down, ACL dropping probe packets from intermediate hops while passing real traffic — intermediate ICMP loss ≠ real fault ; correlate with direct ping to target).

Step 3: Interface & Physical Layer

ip -s link show eth0   # TX/RX errors, drop counters
ethtool eth0           # Speed, Duplex

Sustained growth of rx_dropped / tx_dropped (compare two ip -s link samples) indicates NIC queue overflow or driver drops — these "works but slow/intermittent timeout" faults are often misattributed to peer issues.

Step 4: Align with Change Timeline

If layer 3 "suddenly fully breaks," check last 24 hours for network tickets (switch port ACL changes, VLAN adjustments, cloud security group modifications, BGP route withdrawals). Network-side changes causing "full break" may show clean hop-by-hop probes until a specific hop; mtr loss distribution across affected vs. unaffected subnets is strong evidence of "change impact scope."

Section 3: Firewall & Access Control

Layer 3 works but port unreachable — next stop is access control. Distinguish "which layer drops packets" in this order:

Local iptables/nftables or firewalld

# firewalld environment (common on RHEL)
# List active chains and rules (example)
# target  prot  source  destination
# ACCEPT  tcp   10.20.1.0/24 0.0.0.0/0 multiport dports 3306
firewall-cmd --list-all
iptables -S INPUT | grep 3306   # Bare metal: inspect iptables directly

Most direct way to judge "is rule dropping": examine chain pkts / bytes counters ( iptables -L -n -v): rule pkts growing means traffic matches this rule; target=DROP with growing counter = root cause; pkts=0 means traffic never reached this rule — problem upstream (routing, ACL, peer not sending).

Cloud Security Groups / Network ACLs / Physical Switch ACLs

These three layers are invisible locally; remediation paths differ: cloud security groups viewed in console, changes take effect in seconds to minutes; switch ACLs require network team cooperation. Packet capture comparison ( tcpdump on both source and destination simultaneously) distinguishes "source didn't send" from "destination didn't receive" — dual-end capture with timestamp alignment is the standard technique to demarcate "who drops."

Nginx/Application Listen & Accept

Port "open" but connection immediately refused. Check if target port is listening and by which process:

# Example: check 3306 listener
ss -lntp | grep 3306
# LISTEN 0 151 *:3306 *:* users:(("mysqld",pid=1024,fd=34))

Port not listening (app down, bound to 127.0.0.1 instead of 0.0.0.0 /specific NIC IP) renders "firewall fully open" useless.

Section 4: TCP Connection States & Parameters

First three sections clean, but app reports intermittent timeouts or "connection pool exhausted" — issue at TCP layer.

Connection State Distribution

# View connection states to a target (example output)
# State   Recv-Q  Send-Q   Local-Address:Port   Peer-Address:Port
# ESTAB   0       0        10.20.1.10:43210     10.30.2.50:3306
# TIME_WAIT 0     0        10.20.1.10:43100     10.30.2.50:3306
ss -tn state established '( dport = :3306 or sport = :3306 )' | head
ss -tn state time-wait | wc -l

Abnormally high TIME_WAIT (tens of thousands) indicates short-connection storm (app not reusing connections, new connection per request); new connections fail due to port exhaustion or local port range limits. ESTAB count far exceeding app's configured pool max suggests connection pool leak (connections established but not returned; pool fully occupied, business reports "acquire connection timeout" while underlying connections still hang on peer).

SYN Queue & Accept Queue Overflow

ss -lnt

shows Recv-Q / Send-Q for LISTEN sockets: Send-Q is somaxconn backlog limit; Recv-Q is current accept queue depth. Use nstat for drop counters:

# Example: accept queue overflow counters
nstat -az TcpExtListenDrops TcpExtOverflow

TcpExtListenDrops

growing steadily = accept queue full, new SYNs dropped; client sees "connect sometimes fast, sometimes hangs long" (SYN retransmit). Root cause: app processes connections slower than arrival, or somaxconn set larger than backend can actually consume causing backlog buildup — increasing somaxconn is not a fix; investigate why app processing is slow.

Inspect Individual Slow Connections with ss -i

App reports "specific request unusually slow"; ss -ti shows that connection's rtt, unacked, cwnd:

# Example: connection info to target
ss -ti dst 10.30.2.50
# cwnd:10 rtt:12.34/2.10 unacked:2 reordering:3

rtt

far above network baseline (same subnet sub-millisecond, cross-datacenter few to tens of ms) and continuously rising, combined with growing reordering, indicates link-layer loss/reordering (cross-validate with Section 2 mtr and interface error counters). Small cwnd with high unacked suggests congestion window suppressed (bandwidth congestion or middlebox MTU mismatch triggering Path MTU issues — test with ping -M do -s 1472 binary search for max working size; -M do prohibits fragmentation). If certain size fails while baseline MTU should be 1500, some hop has MTU <1500 and ICMP "fragmentation needed" is firewalled. In containers, Docker overlay default MTU is smaller than host (host 1500 → container ~1450); cross-container communication uses that baseline.

MTU & Fragmentation

Classic symptom: HTTP 200 pages occasionally fail to load, or specific large packets fail while small ones work. ping -M do -s <size> <target> binary search for max working size ( -M do disables fragmentation). If a size starts failing while network baseline MTU is 1500, some hop has MTU <1500 and Path MTU Discovery broken (ICMP "need frag" dropped by firewall). Container environments: Docker overlay network default MTU smaller than host; host 1500 → container ~1450; cross-container traffic judged against that baseline.

Container & Kubernetes Special Considerations

When troubleshooting inside containers, run commands inside the container ( kubectl exec) and understand differences: resolv.conf, /etc/hosts, routing table are injected by runtime; modifications must be made at Deployment config or NetworkPolicy level, not by editing files inside container (lost on restart).

Service ClusterIP is a virtual IP; backend Pods separated by a layer of kube-proxy iptables/IPVS rules. "Pod direct IP works, Service fails" → suspect kube-proxy rules or IPVS session table; inspect ipvsadm -Ln (IPVS mode) or iptables -t filter -L KUBE-* (iptables mode) rule by rule.

NetworkPolicy only restricts Pod-to-Pod east-west traffic and requires CNI support — "NetworkPolicy configured but no effect" first confirm CNI implements the capability, not blame config syntax. ss / netstat inside container show connections in that network namespace; host sees different set. Dual-end packet capture ( tcpdump on Pod network and host eth simultaneously) is standard to demarcate "K8s layer drop" vs "physical layer drop."

Changes, Rollback & Verification

Troubleshooting is read-only; risk comes from remediation changes. Risk-layered approach:

Modify resolv.conf / its options : Backup first cp /etc/resolv.conf /etc/resolv.conf.bak.$(date +%s); verify immediately with dig; revert on anomaly.

Modify firewall rules ( firewall-cmd / iptables): Append rules, don't rewrite entire table; record current ruleset before change ( iptables-save > /tmp/iptables.bak); rollback with iptables-restore < /tmp/iptables.bak. firewalld --permanent vs immediate are two layers — confirm which you're changing and whether restart triggers.

Tune somaxconn / net.core.somaxconn and ulimit : Kernel params are system-wide; sysctl -w immediate but not persistent; sysctl -p persists via sysctl.d files — don't mix; rollback by restoring old values.

Nginx adjust resolver and pool params : After nginx.conf edit, nginx -t syntax check, then nginx -s reload (graceful, no connection drop); never restart causing disconnects. Reload failure (syntax error) leaves Nginx running old config — no "half-new half-old" state.

Cloud security group / switch ACL changes : No direct "rollback command"; screenshot current ruleset before change as recovery reference; after change, "minimal allow verify → gradual allow" canary; on anomaly, restore screenshot rules immediately.

K8s NetworkPolicy / Service config : Before YAML change, kubectl get <resource> -o yaml > backup.yaml; apply new with kubectl apply; on failure, kubectl apply -f backup.yaml.

Unified verification standard: After remediation, "previously failing operation" now succeeds and remains stable for an observation window (at least covering business peak or 15 minutes without recurrence); nstat related counters ( TcpExtListenDrops, TcpRetransSegs) stop growing in post-fix sampling.

Key Judgment Principles to Remember

Converging symptoms (who→who, all/partial, when started, failure mode) matters more than running commands; without them, tool output is uninterpretable.

Troubleshooting order is "near-to-far, app-to-link": DNS → local routing → hop-by-hop mtr → interface/errors → firewall (local

iptables -L -v

pkts

counters judge if rule dropping) → cloud security groups/switch ACLs (dual-end capture demarcation) → TCP state distribution & parameters.

Critical demarcation commands: ip route get (local routing decision), mtr -n (which hop breaks),

iptables -L -n -v

pkts

(rule dropping?),

ss -lnt

Recv-Q

/ Send-Q with nstat -az TcpExtListenDrops (accept queue overflow),

ss -ti

rtt

/ cwnd / unacked (link quality & congestion).

Intermediate hop ICMP loss ≠ real fault; must verify target reachability with real protocol traffic ( dig, curl, nc).

Every remediation is a change: backup before, verify "original failure now works" after, keep recovery artifacts (rule screenshots, YAML backups); cloud/switch scenarios with no rollback command rely on screenshots.

In container/K8s environments, resolv.conf, routing, network namespaces are runtime-injected; troubleshooting commands must run inside Pod; Service adds a kube-proxy layer — "Pod direct IP works, Service fails" prioritize kube-proxy rule inspection.