Understanding Overlay Networks and VxLAN in Cloud Computing

Overlay networks are virtual networks built on top of an existing physical (underlay) network, providing a layer of abstraction that enables flexible networking across data centers and clusters.

The underlay network carries user IP traffic, while the overlay network, similar to a virtual machine, relies on software to create its topology.

In practice, VxLAN (Virtual Extensible LAN) is commonly used to construct overlay networks, allowing two physical hosts to communicate over a three‑layer IP network.

VxLAN uses Virtual Tunnel Endpoints (VTEPs) to encapsulate and decapsulate packets, adding a VxLAN Network Identifier (VNI) and forwarding the payload via UDP.

The packet flow involves the source VM sending an IP packet to its VTEP, which looks up the destination MAC address, constructs a new UDP packet with the VNI, and sends it to the destination VTEP, which then removes the UDP header and forwards the original IP packet to the target VM.

Overlay networks address three key challenges in cloud data centers:

Frequent virtual machine migration across hosts and data centers.

Large‑scale VM deployments that generate massive MAC address tables and ARP traffic.

Limited network isolation capacity of traditional VLANs (max 4096 networks) versus the need for many more virtual networks.

Virtual Machine Migration

When a host fails or requires maintenance, VMs can be moved to another host without changing their IP addresses because the overlay provides a consistent L2 domain across physical machines.

Scale of Virtual Machines

In large Kubernetes clusters (e.g., 5000 nodes), each node may run tens of thousands of containers, causing a huge number of MAC addresses and ARP requests that overwhelm traditional L2 switches.

VxLAN reduces the MAC address table size by keeping MAC learning confined to VTEPs, limiting ARP broadcast to the VTEP layer and significantly lowering the load on core network devices.

Network Isolation

Multi‑tenant data centers require isolation beyond what VLANs can provide; VLANs support only 4096 IDs, whereas VxLAN’s 24‑bit VNI allows up to 16,777,216 virtual networks, satisfying large‑scale tenant isolation needs.

Summary

Overlay networks, typically built with VxLAN, enable seamless VM migration across clusters and data centers, reduce the pressure on L2 switches by limiting MAC and ARP traffic, and provide a vastly larger address space for tenant isolation.

VMs retain connectivity after migration, improving availability and resource utilization.

Network devices handle fewer MAC entries and ARP broadcasts, easing scalability constraints.

The 24‑bit VNI supports up to 16 million virtual networks, facilitating fine‑grained bandwidth allocation and isolation.

While overlay networks solve many challenges, they are not a panacea; other technologies such as NVGRE or STT may be more suitable in certain scenarios.