How to Diagnose Kubernetes Pod Network Issues: Models, Tools, and Real‑World Cases

Pod Network Anomalies

The article classifies pod network problems into several categories:

Network unreachable : ping fails, often caused by firewall rules (iptables, selinux), routing errors, high system load, or link failures.

Port unreachable : ping works but telnet fails, due to firewall restrictions, exhausted connections, or the target application not listening.

DNS resolution failure : domain names cannot be resolved while IP connectivity works, caused by incorrect pod DNS settings, DNS service issues, or communication problems with DNS.

Large packet loss : small packets succeed but large packets are dropped; often related to MTU mismatches (use ping -s to test).

CNI plugin issues : kube-proxy missing iptables/ipvs rules, CIDR exhaustion, or other CNI failures.

These categories are illustrated with a diagram.

Common Network Troubleshooting Tools

tcpdump

tcpdump captures network packets for analysis.

Installation Ubuntu/Debian: apt-get install -y tcpdump CentOS/Fedora: yum install -y tcpdump Alpine: apk add tcpdump --no-cache

Typical commands: tcpdump -i eth0 -nn -w capture.pcap Filters for specific traffic, ports, hosts, protocols, and TCP flags are demonstrated.

nsenter

nsenter allows entering a container’s network namespace from the host to run commands like ifconfig when the container lacks tools.

nsenter -t <pid> -n ifconfig

paping

paping continuously pings a TCP port, useful for testing connectivity and packet loss.

paping -p 80 -c 10 example.com

mtr

mtr combines traceroute and ping to diagnose path latency and loss.

mtr -n google.com

Installation Ubuntu/Debian: apt-get install -y mtr CentOS/Fedora: yum install -y mtr Alpine: apk add mtr --no-cache

Pod Network Troubleshooting Process

The workflow starts from identifying the anomaly type, checking CNI and kube‑proxy status, verifying iptables NAT rules, confirming routing to pod IPs, and finally capturing packets at relevant interfaces (veth, cni0, flannel, host NIC) to locate the failing hop.

Case Studies

1. Service Unreachable After Node Expansion

A newly added node could not reach a ClusterIP service (10.233.0.100:5000) while other nodes could. Investigation showed CNI and kube‑proxy were healthy, but iptables NAT rules correctly translated to the backend pod IP. Packet captures revealed the host’s VXLAN encapsulation used the wrong source IP (10.153.204.228 instead of the node’s actual IP 10.153.204.15) because the NIC had both a static IP and a DHCP‑assigned IP, causing IP conflict. The fix was to disable DHCP on the NIC (set BOOTPROTO="none") and restart Docker and kubelet.

2. External Cloud Host Timeout Accessing Cluster Service

An external VM could telnet to a node‑port but HTTP POST timed out. Wireshark showed the TCP three‑way handshake succeeded, but a 1514‑byte payload was repeatedly retransmitted without ACK. The issue was an MTU mismatch: the VM’s NIC MTU was 1500, while the Calico tunl0 interface used MTU 1440. Reducing the VM’s MTU to 1440 (or increasing Calico’s MTU) resolved the problem.

3. Pod Fails to Access Object Storage by Domain Name

Pods could ping the storage IP but DNS resolution failed, leading to timeouts. The cluster DNS service was functional, but newly created nodes had their kube-proxy pods evicted due to low priority, causing DNS queries to fail. Assigning the system-node-critical priority class to kube-proxy and adding readiness probes to application pods fixed the issue.

References

Original article

paping project