Overlay vs Underlay Networks: VXLAN, MAC VLAN, and Direct Routing Demystified

Overlay Network Model

In physical networks, connecting hosts across multiple bridges can be done by directly linking bridges, placing hosts in the same LAN and ensuring IP addresses do not conflict. For containers, a similar approach uses virtual bridges on each host, assigning each host a distinct subnet to avoid IP conflicts.

Because inter‑host communication must go through external interfaces, a logical network built on top of the physical one—an Overlay network—is required. VXLAN is a popular Overlay tunnel protocol that encapsulates L2 frames in UDP, creating a large “Layer‑2 domain” identified by a VNI. VTEP devices (VXLAN Tunnel Endpoints) perform encapsulation and decapsulation.

VXLAN’s advantages include non‑intrusive use of the underlying network; only VTEP devices need be added. VTEPs can be implemented in hardware switches or in Linux kernels (vxlan module from kernel 3.7). Containers obtain IP addresses from their host’s subnet and communicate via VXLAN tunnels.

Two discovery methods are used: multicast groups or a control‑plane (e.g., etcd) that stores container subnet‑to‑VTEP mappings. Flannel’s VXLAN backend uses the latter.

Underlay Network Model

The Underlay is the traditional physical network of switches and routers that provides data‑plane services for the Overlay. Container Underlay techniques include MAC VLAN, IP VLAN, and direct routing.

MAC VLAN

MAC VLAN creates multiple virtual interfaces on a single physical NIC, each with a unique MAC address. It supports modes such as Private, VEPA, Bridge, and Passthru. Bridge mode is the most common and is the only mode supported by Docker.

IP VLAN

IP VLAN also creates virtual interfaces but shares the physical NIC’s MAC address, avoiding MAC‑spoofing restrictions. It offers L2 (bridge‑like) and L3 (router‑like) models. L2 supports ARP and broadcast; L3 does not support multicast or broadcast.

Linux supports IP VLAN from kernel 4.2, and persistent configuration requires admin scripts.

Direct Routing

Direct routing abandons L2 connectivity across hosts, using L3 routing instead. Solutions like Flannel host‑gw (etcd‑based) and Calico (BGP) dynamically maintain routing tables, allowing containers to communicate via host‑level routes.

Overlay networks add encapsulation overhead but work on most public clouds, while Underlay solutions generally achieve better performance at the cost of stricter underlying network requirements.