Calico BGP vs IPIP: Which Kubernetes Network Mode Delivers Faster Performance?

Overview

The article provides a practical performance comparison of the Calico CNI plugin for Kubernetes, focusing on its two primary network modes: IPIP (overlay) and BGP (layer‑3). It explains Calico’s architecture, the underlying routing mechanisms, and then details a series of latency, bandwidth, and HTTP benchmark tests performed on both physical hosts and pods.

Calico Overview

Calico is a pure L3 network plugin that does not rely on Docker’s docker0 bridge. It implements its own virtual router (vRouter) on each node using Linux kernel routing, assigns an IP address to every container, and distributes routing information via BGP. The plugin also provides DockerDNS, hostname‑based container communication, and flexible network policies based on iptables and ACLs.

Network Modes

Calico supports two deployment modes:

IPIP (overlay) : Each packet is encapsulated in an additional IP header and sent through a tunl0 tunnel device. The tunnel creates an IP‑level bridge between nodes, allowing containers on different hosts to communicate as if they were on the same L2 network.

BGP (layer‑3) : Nodes run a BGP daemon that advertises their pod CIDRs to other nodes. Routing decisions are made by the BGP protocol, eliminating the need for encapsulation and providing higher throughput.

The default calico.yaml enables IPIP; the CALICO_IPV4POOL_IPIP variable can switch it Always (on) or Off (off).

Performance Metrics

The tests measure three key metrics:

Ping latency between hosts and between pods.

Bandwidth using iperf / iperf3.

HTTP throughput with a single‑process Nginx server and ab (ApacheBench).

Physical‑Machine Tests

Ping latency was measured with:

ping master2 | head -n 20 | gawk '/time/ {split($8, ss, "="); sum+=ss[2]; count+=1;} END{print sum/count "ms";}'

Result: average latency 60‑64 ms.

Bandwidth on the host NIC was verified with ethtool ens160, showing a 10 GbE link. iperf tests were run as follows:

# Start server on master2

iperf -s

# Client on master1

iperf -c master2 -i 1

iperf -c master2 -u -t 60 -b 1000M

iperf -c master2 -u -t 60 -b 2000M

Result: average throughput ≈ 4.51 Gbits/s.

HTTP benchmark used a Docker‑run Nginx container:

docker run --name nginx-test -p 8050:80 nginx

ab -n 90000 -c 50 http://172.16.0.1:8050/

Pod‑to‑Pod Tests

Bandwidth between a master node and a pod on node3:

kubectl run iperf3 --image=networkstatic/iperf3 --replicas=3 -- iperf3 -s

iperf3 -c 10.244.5.149 -i 1

Result: ≈ 4.06 Gbits/s.

Bandwidth between pods on node3 and node2:

kubectl exec -it iperf3-57d88c87b4-89kxw -- /bin/sh

iperf3 -c 10.244.4.213 -i 1

Result: ≈ 5.40 Gbits/s.

IPIP Mode Tests

Three scenarios were scripted with a Bash loop (10 iterations) that records sender and receiver bandwidth, then averages the results:

#!/bin/bash
for i in {1..10}; do
  if [ "$i" -eq "1" ]; then
    iperf3 -c master2  > iperf3data
  else
    iperf3 -c master2  >> iperf3data
  fi
done
cat iperf3data | grep sender | awk '{print $7}' > sender
cat iperf3data | grep receiver | awk '{print $7}' > receiver
awk '{sum += $1};END {print "sender Avg= ",sum/NR}' sender
awk '{sum += $1};END {print "receiver Avg= ",sum/NR}' receiver

Results:

Physical‑to‑physical: sender ≈ 5.07 Mbps, receiver ≈ 5.07 Mbps.

Physical‑to‑pod: sender ≈ 4.44 Mbps, receiver ≈ 4.44 Mbps.

Pod‑to‑pod: sender ≈ 4.76 Mbps, receiver ≈ 4.76 Mbps.

Ping latency in IPIP mode:

Host‑to‑host: ≈ 0.189 ms.

Host‑to‑pod: ≈ 0.297 ms.

Pod‑to‑pod: ≈ 0.312 ms.

BGP Mode Tests

The same Bash script was used, but with BGP routing enabled (IPIP disabled). Results:

Physical‑to‑physical: sender ≈ 5.08 Mbps, receiver ≈ 5.08 Mbps.

Physical‑to‑pod: sender ≈ 4.68 Mbps, receiver ≈ 4.68 Mbps.

Pod‑to‑pod across nodes: communication failed (no connectivity).

Ping latency in BGP mode:

Host‑to‑host: ≈ 0.189 ms.

Host‑to‑pod: ≈ 0.268 ms.

Pod‑to‑pod across nodes: no connectivity.

Summary Table

The article includes a comparative table (image) summarizing throughput and latency for both IPIP and BGP modes across the tested scenarios.

Conclusion

Calico’s BGP mode provides significantly higher bandwidth between nodes compared with IPIP, while maintaining low latency. However, BGP mode does not allow cross‑node pod communication when the network policy isolates pods, whereas IPIP supports pod‑to‑pod traffic across nodes. The choice of mode should depend on the required isolation and performance characteristics of the Kubernetes cluster.