Understanding VXLAN: Architecture, Operation, and Deployment in Cloud‑Native Networks

What is VXLAN?

VXLAN (Virtual Extensible LAN) is a widely used network‑virtualization technology that builds a logical tunnel between source and destination devices, encapsulating original Ethernet frames inside UDP packets. The encapsulated packet then travels over a standard IP network, effectively giving virtual machines the ability to communicate across routed networks as if they were on the same Layer‑2 segment.

Why VXLAN is Needed

VXLAN addresses two major trends in modern data‑center environments: the need for live VM migration without changing IP or MAC addresses, and the requirement to isolate a massive number of tenants. Traditional VLANs support only about 4,000 IDs and are limited to small Layer‑2 domains, which cannot satisfy large‑scale cloud deployments.

VM Live Migration

Server virtualization creates multiple VMs on a single physical server, improving utilization and reducing costs. Live migration requires that both IP and MAC addresses remain unchanged and that the VM’s state be preserved, which is only possible within the same Layer‑2 domain. VXLAN extends the Layer‑2 domain across the IP fabric, allowing migration across racks, pods, or even data‑center sites.

Tenant Isolation at Scale

VLANs are limited to roughly 4,000 IDs, insufficient for cloud providers that host tens of thousands of tenants. VXLAN introduces a 24‑bit VXLAN Network Identifier (VNI), supporting up to 16 million isolated segments, thereby meeting large‑scale tenant isolation requirements.

VXLAN vs. VLAN

While VLANs provide limited isolation within a small Layer‑2 broadcast domain, VXLAN offers a 24‑bit VNI for up to 16 M segments and creates a virtual tunnel that turns the underlying IP network into a giant Layer‑2 switch, enabling wide‑area VM migration.

VXLAN Packet Format

The VXLAN encapsulation adds four headers to the original Ethernet frame:

VXLAN Header (8 bytes) – contains the VNI and flags.

UDP Header – source port is derived from a hash of the inner frame; destination port is fixed at 4789.

Outer IP Header – source and destination IPs correspond to the VTEP addresses.

Outer MAC Header – source MAC is the VTEP’s MAC; destination MAC is the next‑hop device’s MAC.

VXLAN Components: VTEP and VNI

VTEP (VXLAN Tunnel Endpoint) is the edge device that encapsulates and decapsulates traffic. VNI (VXLAN Network Identifier) functions like a VLAN ID, uniquely identifying a tenant or broadcast domain. VNIs can be classified as Layer‑2 VNIs for intra‑subnet forwarding or Layer‑3 VNIs associated with VPN instances for inter‑subnet routing.

VXLAN Gateways

VXLAN introduces both Layer‑2 and Layer‑3 gateways. A centralized gateway consolidates all inter‑subnet traffic on a single device, simplifying management but introducing sub‑optimal paths and ARP‑table scaling limits. A distributed gateway places the gateway function on each leaf switch in a spine‑leaf topology, eliminating the ARP bottleneck and providing optimal forwarding.

Establishing VXLAN Tunnels

Tunnels are defined by a pair of VTEP IP addresses. They can be created statically (manual configuration) or dynamically using VXLAN EVPN as the control plane. Dynamic tunnels rely on BGP EVPN to discover VTEPs, exchange VNI and host information, and automatically program forwarding entries.

Determining Tunnel Membership

VXLAN uses Bridge Domains (BD) to group ports, with each BD mapped to a VNI. Broadcast, unknown‑unicast, and multicast (BUM) traffic is replicated to all VTEPs belonging to the same BD.

VXLAN EVPN Control Plane

EVPN (Ethernet VPN) provides a BGP‑based control plane for VXLAN, replacing the manual, flood‑based discovery of the original RFC 7348 design. EVPN advertises VTEP IPs, VNIs, MAC addresses, and IP prefixes using three route types:

Type 2 – MAC/IP route for host discovery.

Type 3 – Inclusive multicast Ethernet tag route for VNI and VTEP discovery.

Type 5 – IP prefix route for routing subnets.

These routes enable automatic VTEP discovery, MAC learning, and reduction of broadcast traffic.

Forwarding in VXLAN Networks

Within a VXLAN domain, Layer‑2 forwarding relies on MAC tables, while Layer‑3 forwarding uses FIB tables, just like traditional networks. In a centralized VXLAN design, intra‑subnet traffic is broadcast via VTEP replication, and inter‑subnet traffic traverses the centralized Layer‑3 gateway. In a distributed design, leaf switches act as both Layer‑2 and Layer‑3 gateways, providing optimal paths and scaling benefits.

For further reading, see the linked “Cloud‑Native Platform” article and the accompanying e‑book collection.