How VXLAN Enables Scalable Data Center Networking and Seamless VM Migration

What is VXLAN

VXLAN (Virtual eXtensible Local Area Network) is an IETF‑defined NVO3 standard that extends traditional VLAN by encapsulating Layer‑2 Ethernet frames inside UDP packets, allowing them to be transported over an IP network.

VXLAN creates a logical tunnel between a source and destination VTEP (VXLAN Tunnel Endpoint), making the data‑center network appear as a giant virtual Layer‑2 switch.

Why VXLAN Is Needed

Data‑center virtualization introduces two critical requirements: seamless VM live migration and isolation for a massive number of tenants. VXLAN satisfies both by providing a scalable Layer‑2 domain over the existing IP fabric.

Server Virtualization

Physical servers typically have low utilization (10‑15%). Virtualization splits a physical server into multiple logical VMs, each with its own OS, MAC, and IP, connected via a virtual switch.

VM Dynamic Migration

Live migration moves a running VM from one physical host to another without service interruption, requiring the VM’s IP and state to remain unchanged. VXLAN’s virtual Layer‑2 domain makes this possible across a large data‑center fabric.

VXLAN vs. VLAN

Traditional VLANs support only ~4,000 IDs, insufficient for large multi‑tenant clouds. VXLAN introduces a 24‑bit VNI (VXLAN Network Identifier), supporting up to 16 million segments, and encapsulates traffic in a tunnel that traverses the IP network.

VXLAN Packet Structure

VXLAN Header (8 bytes) with VNI and flags

UDP Header (destination port 4789)

Outer IP Header (source = local VTEP IP, destination = remote VTEP IP)

Outer MAC Header (source = local VTEP MAC, destination = next‑hop MAC)

Key Concepts: VTEP and VNI

A VTEP is the edge device (hardware switch or virtual switch) that performs VXLAN encapsulation and decapsulation. The VNI identifies a virtual network (tenant) within the VXLAN fabric.

Which VTEPs Need Tunnels

VTEPs that belong to the same Bridge‑Domain (BD) – the VXLAN equivalent of a VLAN – must establish tunnels so that any VM in that BD can communicate, regardless of physical location.

VXLAN Gateways

Two types of gateways enable communication beyond the VXLAN domain:

L2 Gateway : Provides access for endpoints and intra‑VXLAN subnet communication.

L3 Gateway : Handles inter‑subnet traffic and traffic to external networks.

Gateways can be centralized (single device) or distributed (leaf switches in a spine‑leaf architecture). Centralized gateways simplify management but can become bottlenecks; distributed gateways scale better and avoid ARP‑table limits.

Packet Forwarding in a Centralized VXLAN

When a VM sends an ARP request, the local VTEP encapsulates the broadcast, replicates it to all VTEPs in the Peer List, and learns the source MAC/VNI mapping. The destination VM replies with a unicast ARP response, which follows the same encapsulation/decapsulation process. Subsequent traffic uses learned MAC entries for direct unicast forwarding.

Inter‑Subnet Communication

Different subnets reside in separate BDs (e.g., VNI 5000 and VNI 6000). Each BD has a BDIF (Layer‑3 interface) that acts as a gateway. A VM first ARPs for the BDIF MAC, then sends traffic to the BDIF, which routes it to the target subnet, encapsulating the packet into the appropriate VXLAN tunnel.

Communication between VXLAN and non‑VXLAN networks also relies on the L3 gateway, which decapsulates VXLAN traffic before forwarding it to the external network.