Understanding Kubernetes Underlay and Overlay Networks: From Flannel to Calico

This article explores Kubernetes network models, analyzing both underlay and overlay approaches.

Underlay Network Model

What is Underlay Network

Underlay Network refers to the physical network infrastructure such as switches and routers; DWDM links these devices into a physical topology that carries packet traffic.

Underlay can be layer‑2 (e.g., Ethernet) or layer‑3 (e.g., the Internet).

Layer‑2 technologies include VLAN, while layer‑3 technologies include protocols like OSPF and BGP.

Underlay network in Kubernetes

In Kubernetes, a typical underlay example uses the host as a router, with pods learning routing entries to achieve cross‑node communication.

Common implementations are Flannel host‑gw mode and Calico BGP mode.

Flannel host‑gw

In Flannel host‑gw mode each node must reside in the same layer‑2 network, acting as a router; cross‑node traffic is routed via routing tables, effectively simulating an underlay network.

Notes: The cluster CIDR should be at least /16 to ensure the node acts as a layer‑1 network and pods as a separate network; otherwise routing tables may become unreachable.

Calico BGP

BGP (Border Gateway Protocol) is a decentralized autonomous routing protocol that maintains IP routing tables to provide reachability between autonomous systems.

Unlike Flannel, Calico consists of multiple daemons: the BGP client (Felix) obtains routes and distributes them to peers, while a router reflector optimizes internal BGP connections, reducing the number of required BGP sessions.

IPVLAN & MACVLAN

IPVLAN and MACVLAN are NIC virtualization technologies. IPVLAN allows a physical NIC to have multiple IP addresses sharing the same MAC, whereas MACVLAN allows multiple MAC addresses on a single NIC, often without IP addresses.

These techniques flatten the pod network to the node level, offering higher performance and lower latency, and are considered overlay networks.

Virtual bridge : creates a veth pair, one end in the container, the other in the host root namespace, allowing packets to traverse the host network stack.

Multiplexing : uses an intermediate device with multiple virtual interfaces distinguished by MAC/IP.

Hardware switching : assigns a virtual NIC to each pod; modern NICs support SR‑IOV, virtualizing a physical NIC into multiple VF interfaces that share the PCIe lane.

In Kubernetes, typical CNI plugins for IPVLAN are Multus and DANM.

Multus

Multus is an open‑source CNI that combines traditional CNI with an SR‑IOV plugin, enabling pods to connect to SR‑IOV VFs and leveraging IPVLAN/MACVLAN features.

When a pod is created, the SR‑IOV plugin moves the VF into the pod's network namespace, sets the interface name from the CNI configuration, and brings the VF up.

eth0 is the default Flannel network.

VF is an instance of the host physical port (e.g., ens2f0) and appears as south0 inside the pod.

A second VF uses the DPDK driver, appears as north0, and is bound to the vfio‑pci driver.

DANM

DANM is Nokia's open‑source CNI project that brings carrier‑grade networking to Kubernetes, supporting SR‑IOV/DPDK and IPVLAN.

Overlay Network Model

What is Overlay

Overlay networks use virtualization on top of the underlay, employing tunnel protocols to encapsulate packets and create logical networks without altering the physical infrastructure.

Common tunnel technologies

GRE (Generic Routing Encapsulation) : encapsulates IPv4/IPv6 packets within another protocol, operating at L3.

VxLAN (Virtual Extensible LAN) : encapsulates L2 Ethernet frames inside UDP packets (default port 4789), extending VLAN IDs to millions of virtual networks.

Typical overlay implementations include Flannel and Calico VxLAN/IPIP modes.

IPIP

IP in IP is a tunnel protocol similar to VxLAN, implemented via the Linux kernel module ipip.ko.

In Kubernetes, IPIP functions like VxLAN but encapsulates packets within IP packets rather than UDP.

VxLAN

Linux added VxLAN support in kernel 3.7.0; stable operation is recommended on kernels 3.9.0 or newer.

In Kubernetes, Flannel daemons maintain VxLAN devices per node (e.g., flannel.1) using VNIDs and routing tables; cross‑node traffic is forwarded based on learned MAC addresses.

$ bridge fdb
26:5e:87:90:91:fc dev flannel.1 dst 10.0.0.3 self permanent

Notes: VxLAN uses UDP port 4789, while Flannel’s default port is 8472.

Weave VxLAN

Weave also uses VxLAN (called fastdp) built on the Linux openvswitch datapath module and encrypts traffic.

Notes: fastdp operates in kernel ≥3.12; on older kernels (e.g., CentOS 7) it runs in userspace (sleeve mode).