Mastering Kubernetes Networking: Choose the Right CNI Plugin and Boost Performance

Kubernetes Network Model Explained: CNI Plugin Selection and Performance Comparison

"Each Pod has an independent IP, and Pods can communicate directly"

Why Network is Kubernetes's Biggest Pain Point?

As a frontline ops engineer with 5 years experience, I've seen many incidents caused by misconfigured networks:

Midnight emergency: Pods cannot communicate, the whole microservice cluster is down.

Performance nightmare: Network latency spikes to 500 ms, users start calling in.

Scaling shock: After adding new nodes, 30% of Pods experience network errors.

If you have encountered these issues, you're in the right place. Today I will share core principles and practical experience.

What Will You Gain?

✅ Deep understanding of the essence of the K8s network model (beyond concepts).

✅ In‑depth comparison of 7 mainstream CNI plugins (with performance data).

✅ Production‑grade selection decision framework.

✅ Real‑world tuning tricks (hard‑earned experience).

✅ Common issue troubleshooting guide.

Chapter 1: The "Three‑Layer Universe" of Kubernetes Network Model

1.1 Network Model Overview

Kubernetes network design follows a simple yet powerful principle: "Each Pod gets its own IP, and Pods can communicate directly".

┌─────────────────────┐
│    Application Layer │  ← Service/Ingress
├─────────────────────┤
│    Pod Network Layer │  ← Pod‑to‑Pod communication
├─────────────────────┤
│    Node Network Layer │  ← Physical/virtual network
└─────────────────────┘

1.2 Four Communication Scenarios

Scenario 1: Container‑to‑Container

Principle : Shared network namespace.

Implementation : Direct localhost communication.

Performance : Near zero overhead.

Scenario 2: Pod‑to‑Pod Same Node

Principle : Virtual bridge (cbr0/cni0).

Path : Pod A → veth pair → Bridge → veth pair → Pod B.

Latency : Typically < 0.1 ms.

Scenario 3: Pod‑to‑Pod Across Nodes

Core Challenge : Main battlefield for CNI plugins.

Common Solutions : Overlay network, BGP routing, host routing.

Latency Impact : 0.2 ms – 2 ms depending on solution.

Scenario 4: External‑to‑Pod

Implementation : Service + kube‑proxy.

Modes : iptables / ipvs / eBPF.

Considerations : Load‑balancing strategy, session persistence.

Chapter 2: Deep Dive into CNI Plugins and Selection

2.1 Plugin Classification

I categorize mainstream CNI plugins into three groups based on implementation principles:

Overlay Network

Flannel (VXLAN)

Calico (IPIP)

Weave

Routing Network

Calico (BGP)

Kube‑router

High Performance

Cilium (eBPF)

SR‑IOV

2.2 Detailed Comparison

Flannel

# Flannel configuration example
apiVersion: v1
kind: ConfigMap
metadata:
  name: kube-flannel-cfg
data:
  net-conf.json: |
    {
      "Network": "10.244.0.0/16",
      "Backend": {
        "Type": "vxlan",
        "Port": 8472,
        "VNI": 1,
        "DirectRouting": true
      }
    }

Advantages : Simple deployment, community support, quick start.

Deploy in 5 minutes.

Well‑documented, strong community.

Low network environment requirements.

Disadvantages : Moderate performance (encapsulation overhead), limited features.

Average performance.

No network policy support.

Scalability limits for large clusters.

Suitable Scenarios : Testing, small clusters (< 100 nodes).

Test environments.

Small clusters (< 100 nodes).

Complex network environments.

Calico

# Calico configuration example
apiVersion: operator.tigera.io/v1
kind: Installation
metadata:
  name: default
spec:
  calicoNetwork:
    ipPools:
    - blockSize: 26
      cidr: 10.48.0.0/16
      encapsulation: VXLANCrossSubnet
      natOutgoing: Enabled

Advantages : BGP mode offers excellent performance, full network‑policy feature set, supports multiple data planes, proven at large scale.

High performance in BGP mode.

Complete L3/L4 policy support.

Supports many data planes.

Validated in large‑scale clusters.

Disadvantages : Complex configuration, requires BGP support, troubleshooting can be difficult.

High configuration complexity.

Requires BGP network.

Debugging can be hard.

Suitable Scenarios : Production, large clusters (500+ nodes), need network policies, performance‑critical workloads.

Production‑grade choice.

Large clusters (500+ nodes).

Environments requiring network policies.

High‑performance applications.

Cilium

# Cilium configuration example
apiVersion: v1
kind: ConfigMap
metadata:
  name: cilium-config
  namespace: kube-system
data:
  enable-bpf-masquerade: "true"
  enable-xt-socket-fallback: "false"
  install-iptables-rules: "false"
  tunnel: vxlan
  enable-bandwidth-manager: "true"
  enable-local-redirect-policy: "true"

Advantages : eBPF delivers extreme performance, L7 network‑policy support, service‑mesh capabilities, excellent observability.

eBPF technology, top‑tier performance.

L7 network‑policy support.

Service‑mesh capabilities.

Great observability.

Disadvantages : Requires recent kernel (≥ 4.9), newer technology with higher risk, debugging is challenging.

High kernel version requirement.

Relatively new, higher risk.

Debugging difficulty.

Suitable Scenarios : Cloud‑native apps, ultra‑high performance, need L7 policy, strong team expertise.

Cloud‑native applications.

Very high performance requirements.

Need L7 policy control.

Strong technical team.

2.3 Performance Benchmark

Based on production data from a 1000‑node cluster:

CNI Plugin

Pod Startup Time

Network Latency

Bandwidth

CPU Overhead

Memory Overhead

Flannel(VXLAN)

3.2s

0.8ms

8.5 Gbps

Medium

150 MB

Calico(BGP)

2.1s

0.3ms

9.2 Gbps

Low

120 MB

Calico(IPIP)

2.8s

0.6ms

8.8 Gbps

Medium

140 MB

Cilium

2.5s

0.2ms

9.8 Gbps

Low

200 MB

Weave

4.1s

1.2ms

7.8 Gbps

High

180 MB

Key Findings :

Cilium delivers the best latency and bandwidth.

Calico BGP offers excellent overall performance.

Flannel is easy to deploy but performance is average.

Chapter 3: Production Selection Framework

3.1 Decision Tree

Start selection
   ↓
Is it production?
├─ No → Flannel (quick start)
└─ Yes ↓
   Cluster size?
   ├─ <50 nodes → Flannel/Calico
   ├─ 50‑500 nodes → Calico
   └─ >500 nodes ↓
       Performance requirement?
       ├─ Normal → Calico BGP
       └─ Very high → Cilium

3.2 Evaluation Dimensions

Business Characteristics

Application type : CPU‑intensive vs I/O‑intensive.

Traffic pattern : East‑west vs north‑south ratio.

Latency requirement : Real‑time < 5 ms, general < 50 ms.

Bandwidth demand : Peak and burst handling.

Technical Environment

Kubernetes version : 1.20+ recommends Cilium.

Kernel version : Affects eBPF support.

Network environment : BGP support.

Team capability : Operational complexity tolerance.

Cost‑Benefit

Hardware cost : Network devices, server specs.

Manpower cost : Learning, maintenance, incident handling.

Stability risk : New tech vs mature solution.

Chapter 4: Practical Tuning Techniques

4.1 Flannel Optimization

Performance Settings

apiVersion: v1
kind: ConfigMap
metadata:
  name: kube-flannel-cfg
data:
  net-conf.json: |
    {
      "Network": "10.244.0.0/16",
      "Backend": {
        "Type": "vxlan",
        "Port": 8472,
        "VNI": 1,
        "DirectRouting": true
      }
    }

Enable DirectRouting to bypass encapsulation.

Adjust MTU to avoid fragmentation.

Reduce iptables rule count.

Diagnostic Commands

# Check flannel status
kubectl get pods -n kube-system | grep flannel
# View routing table
ip route show
# Inspect vxlan interface
ip link show flannel.1
# Packet capture
tcpdump -i flannel.1 -n

4.2 Calico Deep Tuning

BGP Configuration

apiVersion: projectcalico.org/v3
kind: BGPConfiguration
metadata:
  name: default
spec:
  logSeverityScreen: Info
  nodeToNodeMeshEnabled: false
  asNumber: 64512
  serviceClusterIPs:
  - cidr: 10.96.0.0/12

Network Policy Best Practice

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: deny-all-default
spec:
  podSelector: {}
  policyTypes:
  - Ingress
  - Egress

Production Tuning

# Adjust IP pool block size
calicoctl create -f - <<EOF
apiVersion: projectcalico.org/v3
kind: IPPool
metadata:
  name: default-ipv4-ippool
spec:
  cidr: 10.48.0.0/16
  blockSize: 24
EOF
# Felix configuration
apiVersion: projectcalico.org/v3
kind: FelixConfiguration
metadata:
  name: default
spec:
  bpfLogLevel: ""
  logSeverityFile: Info
  logSeverityScreen: Info
  reportingInterval: 30s
  chainInsertMode: Insert
  defaultEndpointToHostAction: ACCEPT
  iptablesFilterAllowAction: ACCEPT
  iptablesMangleAllowAction: ACCEPT

4.3 Cilium Advanced Settings

eBPF Acceleration

apiVersion: v1
kind: ConfigMap
metadata:
  name: cilium-config
  namespace: kube-system
data:
  enable-bpf-masquerade: "true"
  enable-xt-socket-fallback: "false"
  install-iptables-rules: "false"
  tunnel: vxlan
  enable-bandwidth-manager: "true"
  enable-local-redirect-policy: "true"

Observability

# Enable Hubble UI
cilium hubble enable --ui
# Observe traffic
hubble observe --follow
# List metrics
cilium metrics list

Chapter 5: Common Issues Quick‑Reference

5.1 Connectivity Problems

Issue 1: Pods cannot access external network

Symptoms : Ping from pod to internet fails.

# Check DNS resolution
nslookup google.com
# Check NAT rules
iptables -t nat -L POSTROUTING
# Check routes
ip route show
# Inspect CNI config
cat /etc/cni/net.d/10-flannel.conflist

Common Causes :

NAT rule missing.

Firewall blocking.

CNI configuration error.

Issue 2: Cross‑node pod communication fails

Symptoms : Same‑node pods work, cross‑node pods cannot talk.

# Flannel tunnel check
bridge fdb show dev flannel.1
ping <peer-flannel-ip>
netstat -ulnp | grep 8472
# Calico BGP check
calicoctl node status
calicoctl get ippool -o wide
calicoctl get felixconfiguration

5.2 Performance Diagnosis

High latency

# Basic connectivity test
kubectl run test-pod --image=nicolaka/netshoot -it -- /bin/sh
ping <target-pod-ip>
traceroute <target-pod-ip>
iperf3 -c <target-pod-ip>
# System load
top
iostat
sar -n DEV 1

Low throughput

# Adjust NIC queues
ethtool -L eth0 combined 4
# Kernel parameters
echo 'net.core.rmem_max = 134217728' >> /etc/sysctl.conf
echo 'net.core.wmem_max = 134217728' >> /etc/sysctl.conf
sysctl -p
# Verify CNI MTU
ip link show cni0

5.3 Recovery Procedures

CNI Plugin Recovery

# Restart CNI pods
kubectl delete pod -n kube-system -l app=flannel
# Clean old configs (use cautiously)
rm -rf /var/lib/cni/networks/*
rm -rf /var/lib/cni/results/*
# Re‑install
kubectl apply -f kube-flannel.yml
# Verify
kubectl get pods -o wide

Network Policy Reset

# Remove all policies (emergency)
kubectl delete networkpolicy --all -A
# Re‑apply needed policies
kubectl apply -f network-policies/

Chapter 6: Monitoring and Operations Best Practices

6.1 Key Monitoring Metrics

Basic Network Metrics

# Prometheus alert example
groups:
- name: kubernetes-network
  rules:
  - alert: PodNetworkLatencyHigh
    expr: histogram_quantile(0.95, network_latency_seconds) > 0.1
    for: 2m
    annotations:
      summary: "Pod network latency too high"
  - alert: CNIPodStartSlow
    expr: pod_start_duration_seconds > 30
    for: 1m
    annotations:
      summary: "Pod startup time too long"

CNI‑Specific Metrics

Flannel: VXLAN tunnel status, UDP port health, route consistency.

Calico: BGP session health, Felix component status, IP pool usage.

Cilium: eBPF program load, Hubble flow stats, L7 policy hit rate.

6.2 Log Collection Strategy

Structured Log Config

# Fluent Bit CNI log collection
apiVersion: v1
kind: ConfigMap
metadata:
  name: fluent-bit-config
data:
  fluent-bit.conf: |
    [INPUT]
        Name tail
        Path /var/log/pods/*/*/*.log
        Parser docker
        Tag kube.*
    [FILTER]
        Name grep
        Match kube.*
        Regex log flannel|calico|cilium

6.3 Automation Operations

CNI Health‑Check Script

#!/bin/bash
check_cni_pods() {
  echo "Checking CNI pod status..."
  kubectl get pods -n kube-system -l app=flannel -o wide
  for node in $(kubectl get nodes -o name); do
    echo "Checking node $node CNI config..."
    kubectl describe $node | grep -A 5 "Network Plugin"
  done
}
check_pod_networking() {
  echo "Checking pod network connectivity..."
  kubectl run net-test --image=busybox --restart=Never -- sleep 3600
  kubectl wait --for=condition=ready pod net-test --timeout=60s
  kubectl exec net-test -- ping -c 3 kubernetes.default.svc.cluster.local
  kubectl delete pod net-test
}
main() {
  check_cni_pods
  check_pod_networking
}
main "$@"

Chapter 7: Advanced Topics and Future Trends

7.1 Multi‑CNI Hybrid Deployment

# Example pod using Cilium network
apiVersion: v1
kind: Pod
metadata:
  name: high-performance-pod
  annotations:
    k8s.v1.cni.cncf.io/networks: cilium-net
spec:
  containers:
  - name: app
    image: nginx

7.2 Service Mesh Integration

# Cilium + Istio integration
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
metadata:
  name: control-plane
spec:
  values:
    pilot:
      env:
        EXTERNAL_ISTIOD: false
    cni:
      enabled: true
      provider: cilium

7.3 Cloud‑Native Network Security

# Micro‑segmentation network policy
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: micro-segmentation
spec:
  podSelector:
    matchLabels:
      app: frontend
  policyTypes:
  - Ingress
  - Egress
  ingress:
  - from:
    - podSelector:
        matchLabels:
          app: gateway
    ports:
    - protocol: TCP
      port: 80

Conclusion: Your CNI Selection Path

After in‑depth analysis, here are the practical recommendations:

Top Recommendations

Small Team Quick Start

Flannel → validate → Calico → scale.

Enterprise Production

Calico (BGP) + network policies + Prometheus monitoring.

Performance‑Critical

Cilium (eBPF) + Hubble observability + L7 policies.

Key Decision Points

Start simple unless high performance is required.

Prioritize observability; network issues are hard to debug.

Match technology complexity to team capability.

Evolve in stages, avoid one‑shot migrations.

Action Plan

Phase 1: Basic Setup (1‑2 weeks)

Deploy Flannel, verify basic functionality.

Establish network monitoring.

Define network policy baseline.

Phase 2: Production Ready (3‑4 weeks)

Migrate to Calico, configure BGP.

Implement network policies.

Finalize incident response procedures.

Phase 3: Advanced Features (1‑2 months)

Evaluate Cilium feasibility.

Deploy service mesh integration.

Optimize network performance.