Understanding Overlay vs. Underlay Networks in Kubernetes

The source material, taken from "Kubernetes Advanced Practice (2nd Edition)", introduces two fundamental network models used in Kubernetes container networking: Overlay and Underlay.

Overlay Network Model

In a physical network, hosts on the same LAN can communicate directly, but extending this across multiple hosts requires a logical network built on top of the existing infrastructure, known as an Overlay network. Overlay networks encapsulate container traffic in tunnels, allowing containers to retain their own IP addresses while the underlying hosts only need to support the tunnel protocol.

VXLAN is the most popular Overlay tunnel protocol. It encapsulates L2 frames inside UDP packets (L2‑over‑L4), enabling a large Layer‑2 domain (Bridge‑Domain, BD) identified by a VXLAN Network Identifier (VNI). VXLAN requires a sufficiently large MTU to avoid fragmentation and adds header overhead, which can affect performance.

VXLAN’s key advantage is its non‑intrusive nature: the underlying network does not need to be modified. Administrators add VXLAN Tunnel Endpoints (VTEPs) at the network edge, which handle encapsulation and decapsulation. Linux kernels from version 3.7 include native VXLAN support via the vxlan module, allowing each host to act as a VTEP.

Two common methods for VTEP discovery are:

Multicast: VTEPs join a multicast group to discover each other.

Control‑center (etcd): A central store (e.g., etcd) holds container subnet‑to‑VTEP mappings; Flannel’s default VXLAN backend uses this approach.

Underlay Network Model

Underlay refers to the traditional physical network composed of switches, routers, Ethernet, routing protocols, and VLANs. It provides the data‑plane for Overlay networks and can also be used directly for container networking.

1. MAC VLAN

MAC VLAN creates multiple virtual interfaces on a single physical NIC, each with a unique MAC address. It eliminates the need for bridges or NAT, allowing containers to connect directly to the physical network. MAC VLAN offers four modes:

Private – isolates VLAN instances on the same physical port.

VPEA – allows communication between instances if the upstream switch supports hair‑pin mode.

Bridge – creates a bridge on the physical port, the most common mode and the only one supported by Docker.

Passthru – one instance directly uses the physical NIC.

Except for Passthru, traffic is filtered, providing strong isolation suitable for multi‑tenant environments. However, MAC VLAN requires the NIC to operate in promiscuous mode, which many public clouds disallow.

2. IP VLAN

IP VLAN also creates virtual interfaces but shares a single MAC address across them, avoiding MAC‑spoofing restrictions. It supports L2 (bridge‑like) and L3 (router‑like) models. The L2 mode behaves like MAC VLAN Bridge, while the L3 mode routes traffic without supporting broadcast or multicast.

IP VLAN is preferred when the upstream network enforces MAC‑address security or when many virtual interfaces are needed without enabling promiscuous mode. Linux kernels support IP VLAN from version 4.2, but interfaces created with ip link are not persistent without admin scripts.

3. Direct Routing

Direct routing abandons L2 connectivity across hosts, relying on L3 routing protocols. Each host runs a bridge for local containers, and inter‑host traffic is routed via the host’s physical interface. Solutions include:

Flannel host‑gw, which stores routes in etcd and updates them via the flanneld daemon.

Calico, which uses BGP to distribute routes automatically, creating a pair of veth devices per container.

Direct routing typically offers better performance than Overlay because it avoids extra encapsulation overhead, but it imposes stricter requirements on the underlying network.

Conclusion

Overlay networks (e.g., VXLAN) provide flexibility and minimal changes to the physical infrastructure, at the cost of additional encapsulation overhead. Underlay approaches (MAC VLAN, IP VLAN, direct routing) can deliver higher performance but require more careful configuration of the underlying network and may be limited by cloud provider constraints.