Deep Dive into Calico’s Underlying Communication: BGP, IPIP, and VXLAN Modes

Calico network fundamentals : each pod receives a PodIP/32 address (no subnet), the default gateway is the link‑local address 169.254.1.1, and pods connect to the host via a veth pair whose host‑side interface is named calixxxxxx. Calico forces proxy‑ARP on that interface so that ARP replies make the pod reachable.

BGP mode – pure L3 routing, no encapsulation

PodA builds an ICMP request with source 10.244.0.10 and destination 10.244.1.10, finds no direct route, and forwards the packet to the default gateway 169.254.1.1 via the calixxxxxx interface; proxy‑ARP replies with the host’s MAC, updating PodA’s ARP table.

The packet travels through the veth pair to Node1’s kernel routing stack.

Node1’s routing table, populated by BIRD, contains the route 10.244.1.0/24 via 192.168.2.20 dev eth0 proto bird, so the packet is forwarded toward Node2.

Node1 looks up the ARP table, sends the frame to Node2, where the FEFIX‑generated route delivers it to PodB’s calicoXXX interface, completing the delivery.

Summary : the whole flow uses only L3 routing; source and destination IPs remain the original pod IPs, and TCP bandwidth and latency are almost identical to the underlying physical network.

IPIP mode – L3 tunnel for cross‑subnet traffic

IPIP is used when the underlying network cannot learn pod subnets (e.g., public clouds or multi‑subnet environments). It encapsulates the original pod IP packet inside a host‑level IP packet, adding only a 20‑byte outer IP header.

PodA’s ARP/ARP‑reply steps are identical to BGP.

The packet is handed to the tunl0 virtual tunnel interface, where the Linux kernel adds the outer IP header.

Node1 routes the encapsulated packet out through eth0 to Node2.

Node2’s kernel recognises the IPIP protocol (IP protocol 4), strips the outer header, and restores the original pod packet.

The remaining delivery to PodB mirrors the BGP path.

Summary : IPIP adds only the outer IP header, so performance is close to BGP with a small overhead.

VXLAN mode – L2‑over‑UDP tunnel for highly restricted networks

VXLAN provides the highest compatibility, working when the underlying network blocks IPIP or BGP. It encapsulates the pod’s original Ethernet frame inside a UDP packet, creating a virtual L2 broadcast domain over the IP network.

PodA’s ARP steps are the same as the other modes.

Node1’s routing selects the vxlan.calico tunnel interface (generated by Felix) and hands the packet to the kernel VXLAN module.

The kernel builds a full VXLAN packet: inner Ethernet frame (source MAC = Node1 vxlan.calico, destination MAC = Node2 vxlan.calico), inner IP header (src 10.244.0.10, dst 10.244.1.10), VXLAN header (VNI 4096), outer UDP header (dst 4789), and outer IP header (src 192.168.1.10, dst 192.168.2.20).

Node1 sends the UDP‑VXLAN packet via eth0; Node2 receives it, the VXLAN module strips the outer UDP, VXLAN, and IP headers, and restores the original pod Ethernet frame.

The packet is then delivered to PodB; the return path is symmetric.

Summary : VXLAN works in any environment that allows UDP 4789, but the additional Ethernet, UDP, and VXLAN headers add about 50 bytes of overhead, resulting in 10‑15 % performance loss compared with the other modes.

Side‑by‑side comparison

Core positioning : BGP – high‑performance production; IPIP – balance of compatibility and performance; VXLAN – ultimate compatibility fallback.

Encapsulation : BGP – none; IPIP – L3 IP‑in‑IP; VXLAN – L2 frame + UDP.

Extra overhead : BGP – 0 bytes; IPIP – 20 bytes (outer IP); VXLAN – ~50 bytes (UDP + VXLAN + L2 header).

Performance : BGP – near‑physical; IPIP – <5 % loss; VXLAN – 10‑15 % loss.

Dependencies : BGP – underlying network must support BGP; IPIP – host IP reachable and IPIP allowed; VXLAN – host IP reachable and UDP 4789 allowed.

Same‑subnet optimisation : BGP – native no‑encap; IPIP – CrossSubnet auto no‑encap; VXLAN – DirectRouting auto no‑encap.

Operational complexity : BGP – high (requires BGP knowledge); IPIP – low (out‑of‑the‑box); VXLAN – very low (full‑scene compatibility).

Typical use cases : BGP – same‑room production clusters, low‑latency high‑concurrency workloads; IPIP – private clouds, cross‑subnet clusters, public‑cloud‑compatible scenarios; VXLAN – multi‑cloud clusters, highly restricted enterprise networks, test environments.