Kubernetes Networking: From CNI Basics to Troubleshooting

1 Kubernetes Network Model and CNI Role Definition

Understanding Kubernetes networking starts with three core principles:

Principle 1: Every Pod receives a unique, globally routable IP address, enabling direct Pod‑to‑Pod communication without NAT.

Principle 2: Containers in the same Pod share a network namespace, allowing localhost communication.

Principle 3: The node’s Kubelet agent ensures inter‑Pod connectivity; the actual cross‑node implementation is delegated to a CNI plugin.

Kubernetes network communication matrix:

Pod‑internal:  localhost (no CNI involvement)
Pod‑to‑Pod same node: veth pair → bridge (cni0/br0)
Pod‑to‑Pod cross node: veth pair → local cni0 → routing/encapsulation → remote cni0 → remote Pod
Pod → external: NAT via Node IP + SNAT (default CNI behavior)
External → Pod: Service → NodePort / Ingress → Pod

CNI (Container Network Interface) is the standard interface between Kubernetes and network plugins, defined by CoreOS in 2016 and donated to CNCF in 2019. It specifies three operations— ADD (create network), DEL (delete network), and CHECK (verify status)—which Kubelet invokes for each Pod.

In production environments (2026), the dominant CNI plugins are:

Calico : excels at NetworkPolicy, suited for high‑security clusters.

Flannel : simple, zero‑configuration, ideal for quick start‑ups.

Cilium : leverages eBPF for performance and security, a 2026 technology hotspot.

2 Main CNI Plugin Comparison

2.1 Flannel: Simple Overlay Network

Flannel, originally from CoreOS, follows a "simple enough" philosophy.

Working principle : each node gets a /24 subnet; Pod IPs are allocated from this subnet. Cross‑node traffic is encapsulated in VXLAN or UDP overlay.

Flannel cross‑node communication path:

Pod‑A (10.244.0.15) → eth0 → veth pair → cni0 (10.244.0.1)
  → kernel routing lookup → target IP 10.244.2.30 not local
  → encapsulate into VXLAN (outer IP = Node‑2 IP 10.112.0.52)
  → forward over physical network to Node‑2
  → Node‑2 decapsulates VXLAN
  → send to cni0 → veth pair → Pod‑B (10.244.2.30)

VXLAN encapsulation : Flannel creates a flannel.1 VXLAN device on each host; original Ethernet frames are wrapped in UDP (port 4789) with the remote host IP as the outer destination.

Advantages : works out‑of‑the‑box with almost zero configuration; switching between UDP (deprecated) and VXLAN only requires editing the networking/backend setting.

Limitations : no NetworkPolicy support; overlay adds ~5‑10 % performance overhead; in 2026 Flannel is mainly used for rapid validation and small clusters.

2.2 Calico: Routing with BGP and NetworkPolicy

Calico (Tigera) is a leading CNI in 2026. Unlike Flannel’s overlay, Calico typically uses BGP to advertise Pod subnets, allowing direct routing without encapsulation.

BGP routing mode :

Node‑1 (IP: 10.112.0.51, subnet: 10.244.0.0/24)
Node‑2 (IP: 10.112.0.52, subnet: 10.244.2.0/24)

Node‑1 advertises via BGP: 10.244.2.0/24 via 10.112.0.52
Physical switch propagates the route.
When a Pod on Node‑1 contacts a Pod on Node‑2:
  - Source IP: 10.244.0.15
  - Destination IP: 10.244.2.30
  - Physical router forwards directly to Node‑2 (no NAT, no VXLAN).

Felix runs on each node to configure routes, ACLs, and iptables rules locally.

BIRD is the BGP daemon handling peer negotiations and route exchanges.

Typha reduces load on the etcd datastore by multiplexing Felix’s requests when the cluster exceeds ~50 nodes.

# Calico installation (2026 3.28.x)
apiVersion: operator.tigera.io/v1
kind: Installation
metadata:
  name: default
spec:
  calicoNetwork:
    bgp: Enabled
    ipPools:
    - name: default-ipv4-pool
      cidr: 10.244.0.0/16
      natOutgoing: Enabled
      blockSize: 26
    encapsulation: VXLANCrossSubnet
    nodeMetricsPort: 9091
    typha:
      enabled: true
      count: 3

Calico NetworkPolicy provides declarative policies that can span namespaces and nodes, supporting richer match criteria (ICMP types, DSCP, direction).

# Calico NetworkPolicy example: restrict DB access to API server
apiVersion: projectcalico.org/v3
kind: NetworkPolicy
metadata:
  name: db-access-policy
  namespace: production
spec:
  selector: app == 'mysql'
  types:
  - Ingress
  ingress:
  - action: Allow
    source:
      selector: app == 'api-server' && role == 'backend'
    protocol: TCP
    destination:
      ports:
      - 3306
    action: Log
    source: {}
  - action: Deny

2.3 Cilium: eBPF‑Based Networking

Cilium (2026) re‑implements Kubernetes networking using eBPF, delivering major gains in performance, security, and observability.

eBPF introduction : a virtual machine in the Linux 4.x kernel that runs sandboxed bytecode. Traditional plugins rely on iptables, whose rule lookup degrades to O(n) when rule count exceeds ~10 000, causing latency spikes. eBPF provides O(1) lookups.

Cilium operation :

Packet path in Cilium:
Pod → veth pair → kernel stack → eBPF tc hook →
  - endpoint map (local pod forwarding) OR
  - routing map (cross‑node lookup) OR
  - identity map (policy enforcement)
If policy matches, forward; otherwise drop.

Core data structures are Endpoint (Pod network endpoint) and Identity (security identity). Policies are based on identity, avoiding issues when Pods are recreated and receive new IPs.

Cilium advantages :

L7 policies : HTTP method/path, gRPC method control beyond L3/L4.

Bandwidth limits : per‑Pod egress shaping with eBPF, more precise than traditional tc.

Service topology awareness : egress-cached-svc bypasses kube‑proxy hot paths, improving performance by >30 %.

Transparent encryption : WireGuard encrypts intra‑Pod traffic without application changes.

# Cilium installation (2026)
apiVersion: cilium.io/v1alpha1
kind: CiliumConfig
metadata:
  name: cilium-config
spec:
  ipam:
    mode: cluster-pool
  operator:
    clusterPoolIPv4PodCIDRList: [10.244.0.0/16]
    clusterPoolIPv4MaskSize: 26
  eBPF:
    enabled: true
  lbMode: snat
  hostRouting: true
  bandwidthManager:
    enabled: true
  encryption:
    enabled: true
    type: WireGuard
  hubble:
    enabled: true
  relay:
    enabled: true
  ui:
    enabled: true

3 Pod Network Communication Path Analysis

3.1 veth Pair and Bridge

When a Pod is scheduled, Kubelet invokes the CNI plugin, which creates a vethXXXX pair: one end attaches to the host bridge (e.g., cni0), the other is renamed eth0 inside the Pod’s network namespace.

# Host view of network devices:
ens160 (physical NIC)
  ↑
cni0 (bridge, 10.244.0.1/24)
  ├─ veth1a2b3c4d → eth0 @ pod-nginx-abc123 (10.244.0.15)
  ├─ veth5d6e7f8g → eth0 @ pod-api-def456 (10.244.0.16)
  └─ veth9h0i1j2k → eth0 @ pod-db-ghi789 (10.244.0.17)
/proc/sys/net/ipv4/ip_forward = 1  # must be enabled

Bridge forwarding principle : a Linux bridge works at Layer 2 like a physical switch, forwarding frames based on MAC address tables, which are static for veth devices.

iptables rules : most CNI plugins still rely on iptables for NAT and filtering, though eBPF plugins can bypass them.

# View KUBE‑SERVICES chain (Service NAT rules)
iptables -t nat -L KUBE‑SERVICES -n --line | head -30
# View KUBE‑NODE‑PORT chain
iptables -t nat -L KUBE‑NODE‑PORT -n --line
# View FORWARD rules added by CNI plugins
iptables -L FORWARD -n | grep -i calico/flannel/cni

3.2 Two Modes for Cross‑Node Pod Communication

Overlay mode (encapsulation) : Flannel VXLAN, Calico VXLAN, etc., encapsulate packets before sending them over the physical network. Advantage: works on any underlying network; drawback: extra overhead.

Routing‑through‑BGP mode : Calico advertises Pod subnets via BGP; packets travel directly on the physical network without encapsulation, offering near‑native performance.

# iperf3 benchmark (dual 10 G nodes)
Scenario: cross‑node Pod‑to‑Pod TCP bandwidth
Overlay (VXLAN): 8.2 Gbps (≈18 % overhead)
Routing‑through‑BGP: 9.6 Gbps (≈4 % overhead)
Physical baseline: 9.8 Gbps
Conclusion: BGP routing outperforms overlay when the network supports BGP.

4 Service and ClusterIP Mechanics

4.1 kube‑proxy with iptables and IPVS

kube‑proxy runs on every node, watches Service and Endpoint objects, and updates local load‑balancing rules.

In 2026 kube‑proxy supports two modes:

iptables mode (default): each Service creates multiple DNAT rules. Linear rule lookup becomes a latency problem when Services exceed ~500.

IPVS mode : a Layer 4 load balancer in the Linux kernel using hash tables for O(1) lookup, providing stable performance at large scale.

# Switch kube‑proxy to IPVS
kubectl edit configmap -n kube-system kube-proxy
# change mode: "" to mode: "ipvs"

# Verify IPVS rules
ipvsadm -L -n
# Expected snippet:
# TCP 10.96.0.1:443  rr 12 0
#   -> 10.112.0.51:6443  Masq 1 0
# TCP 10.96.0.10:53  rr 8 0
#   -> 10.244.0.15:53   Masq 0 4

IPVS scheduling algorithms include round‑robin, weighted round‑robin, least‑connection, etc. For long‑lived connections (e.g., gRPC), the least_conn strategy is recommended.

4.2 ClusterIP Service Discovery

ClusterIP provides a virtual IP reachable only inside the cluster. Pods resolve Service names via CoreDNS.

# DNS resolution flow for http://order-service.production.svc.cluster.local
Pod → glibc nsswitch → nscd → CoreDNS
1. Pod queries the name.
2. /etc/resolv.conf points to kube‑dns.
3. CoreDNS returns the Service's ClusterIP (e.g., 10.96.x.x).
4. Application connects to ClusterIP → iptables/IPVS → DNAT to Pod IP:Port.

CoreDNS is the default DNS server since K8s 1.13. It watches Service and Endpoint changes via the Kubernetes API.

# CoreDNS ConfigMap (coredns)
apiVersion: v1
kind: ConfigMap
metadata:
  name: coredns
  namespace: kube-system
data:
  Corefile: |
    .:53 {
      errors
      health {
        laminated CUE
      }
      ready
      kubernetes cluster.local in-addr.arpa ip6.arpa {
        pods verified
        fallthrough in-addr.arpa ip6.arpa
      }
      prometheus :9153
      forward . 10.112.0.1  # upstream DNS
      cache 30
      loop
      reload
      loadbalance
    }

5 Ingress and NodePort Mechanisms

5.1 Ingress Controller Architecture

Ingress is a L7 entry point defined by an API object; the actual proxy is provided by an Ingress Controller (e.g., Nginx, Traefik, cloud ALB/CLB).

# Ingress request flow
External request → Cloud LB / NodePort → Ingress Controller Pod
  → Nginx/Traefik routes according to Ingress rules
  → Service (ClusterIP) → kube‑proxy → Pod

Nginx Ingress Controller reloads configuration with nginx -s reload, avoiding process restarts and connection drops. Config is stored in a ConfigMap as nginx.conf.

5.2 NodePort and HostNetwork

# NodePort Service example
apiVersion: v1
kind: Service
metadata:
  name: api-service
  namespace: production
spec:
  type: NodePort
  selector:
    app: api
  ports:
  - name: http
    port: 80   # ClusterIP port
    targetPort: 8080   # Pod port
    nodePort: 30080   # NodePort (30000‑32767)
  - name: grpc
    port: 9090
    targetPort: 9090
    nodePort: 30090

HostNetwork mode : Pods share the host’s network namespace ( hostNetwork: true), eliminating CNI involvement but increasing port‑collision risk and reducing isolation.

6 NetworkPolicy Practice

6.1 Namespace‑level Isolation

# Default deny all inbound traffic (except system components)
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-ingress
  namespace: production
spec:
  podSelector: {}
  policyTypes:
  - Ingress
---
# Allow same‑namespace Pods with matching label
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-same-namespace
  namespace: production
spec:
  podSelector:
    matchLabels:
      role: backend
  policyTypes:
  - Ingress
  ingress:
  - from:
    - podSelector:
        matchLabels:
          role: backend

6.2 Calico GlobalNetworkPolicy Across Namespaces

# GlobalNetworkPolicy: allow monitoring namespace to scrape Prometheus endpoints
apiVersion: projectcalico.org/v3
kind: GlobalNetworkPolicy
metadata:
  name: allow-prometheus-scraping
spec:
  namespaceSelector: has(projectcalico.org/name)
  order: 50
  ingress:
  - action: Allow
    protocol: TCP
    destination:
      ports:
      - 9090
      - 9100
    source:
      namespaceSelector: name == "monitoring"
      selector: app == 'prometheus'
  egress:
  - action: Allow

7 DNS Troubleshooting

7.1 DNS Debug Command Chain

# 1. Test DNS inside a pod
kubectl exec -it pod-test -- /bin/sh
nslookup kubernetes.default   # expect Name: kubernetes.default  Address: 10.96.0.1

dig +short kubernetes.default.svc.cluster.local

# 2. View pod DNS configuration
cat /etc/resolv.conf   # expect nameserver 10.96.0.10, search paths, ndots:5

# 3. Check CoreDNS logs for errors or timeouts
kubectl logs -n kube-system -l k8s-app=kube-dns --tail=200 | grep -i error

# 4. CoreDNS health check
kubectl exec -n kube-system -l k8s-app=kube-dns -c coredns -- coredns -plugins

7.2 ndots Configuration Issue

# ndots meaning: if a query contains fewer dots than ndots, the resolver appends each search domain.
# Example ndots=5, query "prometheus-server":
# 1. prometheus-server.production.svc.cluster.local (fail)
# 2. prometheus-server.svc.cluster.local (fail)
# 3. prometheus-server.cluster.local (fail)
# 4. prometheus-server (success)
# Excessive search traversal degrades internal Service lookup performance.

Solution: use fully‑qualified domain names for intra‑cluster services or lower ndots (e.g., to 2‑3) in the pod’s DNS configuration.

# Adjust pod DNS
spec:
  dnsPolicy: ClusterFirstWithHostNet
  dnsConfig:
    nameservers:
    - 10.96.0.10
    searches:
    - production.svc.cluster.local
    - svc.cluster.local
    - cluster.local
    options:
    - name: ndots
      value: "2"
    - name: timeout
      value: "2"
    - name: attempts
      value: "2"

8 Cross‑Node Pod Communication Failure Cases

8.1 Case 1: Calico BGP Neighbor Failure

Symptom : cross‑node Pod cannot be reached, same‑node works.

# Step 1: Check Calico node status
calicoctl node status   # expect BGP state "Established"

# Step 2: Verify routing table for Pod subnets
ip route | grep 10.244   # expect routes via bird for each node subnet

# Step 3: Inspect BIRD logs
kubectl logs -n calico-system -l k8s-app=calico-node --tail=50 | grep -i bgp

# Step 4: Test connectivity between node gateways
ping -I 10.244.0.1 10.244.2.1

Root cause : physical firewall blocks BGP port TCP 179, preventing BGP session establishment.

Fix : open TCP 179 on the network devices and ensure net.ipv4.ip_forward=1 is enabled.

8.2 Case 2: Flannel VXLAN Packet Loss

Symptom : same‑node communication works; cross‑node shows packet loss and jitter.

# Step 1: Verify VXLAN device status
ip -d link show flannel.1   # expect UP, VxLAN id 1, local IP, dstport 8472

# Step 2: Check forwarding database (FDB)
bridge fdb show | grep flannel.1   # should list remote MAC‑IP mappings

# Step 3: Verify MTU settings (VXLAN adds ~50 bytes overhead)
# Physical network MTU = 1500 → flannel MTU should be 1450
ip link set flannel.1 mtu 1450

# Step 4: Inspect loss counters
cat /sys/class/net/flannel.1/statistics/rx_errors
cat /sys/class/net/flannel.1/statistics/tx_dropped

Root cause : physical network uses jumbo frames (MTU 9000) but the VXLAN device kept the default MTU, causing fragmentation and loss.

9 Cilium eBPF New Features

9.1 kube‑proxy Replacement

Cilium can replace kube‑proxy entirely; Service load‑balancing runs in eBPF programs inside the kernel.

# Verify replacement status
kubectl exec -it -n kube-system ds/cilium -- cilium-dbg status | grep KubeProxyReplacement
# Expected output includes "Kube-proxy replacement: enabled"
# Confirm iptables no longer contains KUBE‑SERVICES rules
iptables -t nat -L | grep KUBE   # should be empty or minimal

Benchmark (Cilium official, 2026):

Metric                     kube-proxy (iptables)   Cilium eBPF
Service connection latency          15 µs                8 µs
Max Services without degradation   ~500               ~10 000
P99 latency at 1000 Services       3 ms                0.5 ms

9.2 Hubble: Built‑in Observability

Hubble provides L7 flow visibility and a UI for real‑time Service topology.

# Enable Hubble UI
cilium hubble enable --ui

# Observe traffic from the API server
cilium hubble observe --from-label app=api-server
# Expected rows:
# TIMESTAMP  SOURCE                DESTINATION          TYPE      VERDICT
# 10:23:45   api-server:8080       order-svc:80          HTTP/GET  FORWARDED
# 10:23:46   order-svc:80          mysql:3306            L4/TCP    FORWARDED
# 10:23:47   api-server:8080       redis:6379            HTTP/GET  DENIED

Hubble UI visualizes Service dependencies without needing external tracing systems.

10 Conclusion

The article builds a complete picture of Kubernetes networking, from the three‑principle model to CNI selection, Service mechanics, DNS, Ingress, NetworkPolicy, and real‑world troubleshooting.

CNI selection evidence : Flannel is easy to start but lacks NetworkPolicy; Calico’s BGP routing delivers near‑physical performance (9.6 Gbps vs 9.8 Gbps baseline) and advanced policies; Cilium’s eBPF sustains >10 000 Services with O(1) lookup, making it the 2026 choice for large clusters.

DNS root‑cause evidence : Over 70 % of DNS issues stem from mis‑configured ndots, 20 % from CoreDNS resource exhaustion, and 10 % from upstream DNS failures. Adjusting ndots to 2‑3 yields immediate improvements.

Cross‑node communication evidence : In BGP mode, a blocked TCP 179 firewall prevents pod‑to‑pod traffic; in VXLAN mode, MTU mismatch causes packet loss.

Cilium eBPF advantages evidence : Direct kernel load‑balancing cuts Service latency from 15 µs to 8 µs and reduces 1000‑Service P99 latency from 3 ms to 0.5 ms, as shown in official benchmarks.