How to Diagnose and Fix Pod Network Issues in Kubernetes Clusters

This article introduces a systematic approach for troubleshooting Kubernetes pod network anomalies, covering a network‑exception model, common tools, and several real‑world case studies.

1. Pod Network Exceptions

Network anomalies are divided into six categories:

Network unreachable : ping fails, often caused by firewall rules, routing errors, overloaded hosts, or link failures.

Port unreachable : ping works but telnet to the port fails, usually due to firewall blocks, high system load, or the application not listening.

DNS resolution failure : domain names cannot be resolved while IP connectivity works, caused by incorrect DNS settings or DNS service issues.

Large‑packet loss : small packets succeed but large packets are dropped; test with ping -s and check MTU mismatches.

CNI plugin issues : node can communicate but pods cannot reach cluster addresses, often due to kube‑proxy or CIDR allocation problems.

Other CNI plugin problems

The classification diagram is shown below:

2. Common Network Diagnostic Tools

tcpdump captures packets on an interface. Installation commands:

apt-get install -y tcpdump   # Ubuntu/Debian
yum install -y tcpdump       # CentOS/Fedora
apk add tcpdump --no-cache   # Alpine

Typical usages:

tcpdump -D
tcpdump host 1.1.1.1
tcpdump src|dst 1.1.1.1
tcpdump net 1.2.3.0/24
tcpdump -c 1 -X icmp
tcpdump port 3389
tcpdump portrange 21-23
tcpdump less 32
tcpdump greater 64
tcpdump -i eth0 -nn host 220.181.57.216 and 10.0.0.1
tcpdump -ttnnvvS -i eth0
tcpdump -nnvvS src 10.5.2.3 and dst port 3389
tcpdump dst 192.168.0.2 and src net and not icmp
tcpdump -vv src mars and not dst port 22
tcpdump 'src 10.0.2.4 and (dst port 3389 or 22)'

Filtering TCP flags:

tcpdump 'tcp[13] & 4!=0'   # RST
tcpdump 'tcp[13] & 2!=0'   # SYN
tcpdump 'tcp[13] & 1!=0'   # FIN

Other useful commands:

wireshark : trace streams for detailed analysis.

nsenter : enter a container’s network namespace from the host. Example: nsenter -t 30858 -n ifconfig paping : TCP‑based continuous ping for port testing. Install dependencies on RedHat/CentOS ( yum install -y libstdc++.i686 glibc.i686) or Ubuntu/Debian (no extra deps). Basic usage: paping -p 80 -c 5 example.com mtr : combines traceroute and ping. Example:

mtr -c5 -n google.com

3. Pod Network Troubleshooting Process

The workflow follows the diagram below:

4. Case Studies

Case 1 – Service Unreachable After Node Expansion

After adding a new node, the node could not reach the ClusterIP of a registry service, while other nodes worked fine. Investigation steps:

Verified CNI plugin (flannel vxlan) and kube‑proxy (iptables) were healthy.

Checked iptables NAT rules; they were correct.

Examined routes to the pod IP (10.233.65.46) – they existed.

Captured packets on the sending node and the pod’s host; the sending node repeatedly retransmitted packets, and the receiving node never replied.

Found that the problematic node had two IP addresses on the same NIC (static + DHCP), causing IP conflict and broken VXLAN encapsulation.

Resolution: removed the DHCP configuration (set BOOTPROTO="none") and restarted Docker and kubelet.

Case 2 – External Cloud Host Times Out Accessing In‑Cluster Application

NodePort service was reachable via telnet, but HTTP requests timed out. Packet capture showed the three‑way TCP handshake succeeded, but large packets (1514 bytes) were continuously retransmitted. The MTU on the cloud host was 1500, while the Calico tunnel interface (tunl0) used MTU 1440, causing fragmentation loss.

Resolution: aligned MTU values by setting the host’s NIC MTU to 1440 or adjusting Calico’s MTU to 1500.

Case 3 – Pods Cannot Reach Object Storage by DNS

Pods could ping the storage IP but failed DNS resolution. Investigation revealed that newly created nodes had pending kube‑proxy pods because they lacked the highest priority class, leading to eviction under resource pressure. Consequently, DNS service traffic was blocked.

Resolution: set kube‑proxy priorityClassName to system-node-critical and added readiness probes to application pods.

These examples illustrate how combining systematic classification, appropriate tooling, and careful packet analysis can quickly isolate and resolve Kubernetes network problems.