How VXLAN Overlays Solve Modern Cloud Data Center Challenges

Background

With the development of cloud computing and virtualization, traditional networks cannot meet the scale and flexibility requirements of modern cloud data centers, leading to the concept of overlay networks. VXLAN is widely used in overlay networks. First, we examine the challenges faced by traditional networks as cloud computing evolves.

1. VM migration limited by network architecture

VM migration requires that IP and MAC addresses remain unchanged, which means migration must occur within the same Layer‑2 domain. Traditional networks need multi‑path redundancy and reliability, but spanning‑tree (STP) is complex and does not scale. Proprietary virtualization technologies (e.g., IRF/vPC) simplify topology but only suit small networks and impose strict topology constraints. Larger solutions like TRILL, SPB, FabricPath, VPLS require hardware upgrades, raising deployment costs.

2. VM scale limited by MAC table size

In large Layer‑2 networks, MAC address tables limit the number of VMs. Low‑cost access devices have small tables, reducing the number of VMs that can be supported. Using multiple gateways to share the load can alleviate pressure but increases cost.

3. Network isolation limitations

VLAN provides only 12‑bit identifiers, allowing roughly 4,000 VLANs, which is insufficient for large public‑cloud environments with tens of thousands of tenants.

VXLAN Basics

1. VXLAN Network Model

New elements introduced by VXLAN:

VTEP (VXLAN Tunnel Endpoint)

VTEP can be a physical switch (e.g., Huawei CE series) or a server hosting VMs. It is the endpoint of the VXLAN tunnel and handles encapsulation and decapsulation of VXLAN packets.

VNI (VXLAN Network Identifier)

VNI provides 24‑bit tenant identifiers, supporting up to 16 million isolated tenants, far exceeding VLAN’s capacity.

VXLAN Tunnel

The tunnel is a logical concept similar to GRE, encapsulating original Ethernet frames inside UDP packets for transport over an IP network.

2. How VXLAN Solves the Challenges

2.1 Solving VM migration limits

Overlay networks extend Layer‑2 domains over a Layer‑3 routed fabric, allowing VM migration across physical boundaries without breaking connectivity.

2.2 Solving MAC‑table size limits

VTEP encapsulates VM traffic in UDP, using the physical network’s IP and MAC addresses, so upstream devices do not need to learn each VM’s MAC.

2.3 Solving isolation limits

Each VNI allocates 24 bits, enabling up to 16 million isolated tenants, which resolves VLAN’s scalability issue.

3. VXLAN Packet Format

VXLAN encapsulation adds an 8‑byte VXLAN header (including a 24‑bit VNI), a UDP header (destination port 4789), an outer IP header, and an outer Ethernet header.

VXLAN Forwarding Mechanism

Example using Huawei CE series switches

3.1 Establishing VXLAN Tunnels

In VXLAN, a Bridge‑Domain (BD) corresponds to a VNI. The mapping between BD and VNI is configured on VTEP.

bridge-domain 10   // create BD 10
vxlan vni 5000    // associate VNI 5000 with BD 10

Display command shows the mapping:

<HUAWEI> display vxlan vni
Number of vxlan vni : 1
VNI            BD-ID            State
----------------------------------
5000           10               up

VTEP determines which BD a packet belongs to based on the configuration of its Layer‑2 sub‑interface. Three encapsulation types are supported: dot1q, untag, and default. The default type allows all traffic, while dot1q and untag restrict traffic based on VLAN tags.

Only one type of sub‑interface can coexist on the same physical port; default cannot coexist with dot1q or untag because of ambiguous tunnel selection.

General Tunnel Creation Methods

Manual method

Manually specify source and destination IPs for the VXLAN tunnel.

interface Nve1
 source 1.1.1.1
 vni 5000 head-end peer-list 2.2.2.2
 vni 5000 head-end peer-list 2.2.2.3

The peer list forms the “head‑end replication list” used for broadcast, unknown‑unicast, and multicast (BUM) traffic.

Automatic method

VXLAN tunnels can be auto‑established using protocols such as BGP, typically in EVN or distributed gateway scenarios.

3.2 VXLAN Packet Forwarding Process

Same‑subnet communication

VM_A sends an ARP request; VTEP learns MAC_A and VNI, replicates the packet to all peers, and each remote VTEP learns the MAC after decapsulation. Subsequent traffic uses unicast.

Different‑subnet communication

Inter‑subnet traffic is routed via VTEP’s Bridge‑Domain interfaces (BDIF), which act like VLANIFs, performing ARP to the gateway and then forwarding encapsulated packets across the VXLAN fabric.

VXLAN Deployment Scenarios

Typical spine‑leaf data‑center topology is used.

4.1 Subnet Interconnect

4.1.1 Same‑subnet

Leaf switches act as VTEPs; a VXLAN tunnel between leaves enables intra‑department VM communication while the spine forwards only the encapsulated traffic.

4.1.2 Different‑subnet (centralized gateway)

Spine hosts the VXLAN Layer‑3 gateway; leaves tunnel to spine, enabling cross‑department communication.

4.1.3 Different‑subnet (distributed gateway)

Deploy the VXLAN Layer‑3 gateway on each leaf to avoid spine bottlenecks and reduce bandwidth consumption.

Leaf‑based gateways learn only local tenant entries, eliminating the table‑size bottleneck of centralized gateways.

4.2 VXLAN Reliability

Single‑uplink connections cannot meet reliability requirements. Adding redundant links and using stacking (CSS) at the access layer improves fault tolerance.

4.3 Access‑Layer Reliability

Stacking multiple switches into a virtual device provides redundancy and load‑balancing via Eth‑Trunk.

Conclusion

If you found this article helpful, feel free to click “Read Later” or “Share”. Your support encourages further sharing.