Mastering VXLAN: Principles, Linux & Open vSwitch Implementation Guide

VXLAN Basic Principles

VXLAN (Virtual eXtensible LAN) extends the 12‑bit VLAN identifier to a 24‑bit VNI, allowing up to 16,777,216 logical networks. It encapsulates Ethernet frames inside UDP packets, creating an overlay network on top of an IP fabric, but adds a 50‑byte overhead that may require MTU adjustments.

The encapsulation and decapsulation are performed by VTEP (VXLAN Tunnel End Point) devices, which can be hardware or software based.

VXLAN Communication Process

VXLAN uses IP multicast for BUM (Broadcast, Unknown‑unicast, Multicast) traffic. Unknown‑unicast packets trigger learning mechanisms: the source VTEP floods the packet via multicast, the destination VTEP learns the MAC‑VNI‑VTEP mapping and updates its forwarding table, enabling subsequent unicast delivery.

Open vSwitch (OVS) does not implement IP multicast; instead it emulates flooding with multiple unicast tunnels and can reduce flooding by populating the forwarding table via a controller.

Linux Implementation

Two common Linux VXLAN implementations are the kernel module and Open vSwitch. The following example uses three hosts (192.168.33.12, .13, .14) each with a Linux bridge (br0) and IPs 10.1.1.2/24, 10.1.1.3/24, 10.1.1.4/24.

brctl addbr br0

ip addr add 10.1.1.2/24 dev br0
ip link set up br0

After creating the bridge, a VXLAN interface is added:

ip link add vxlan0 type vxlan id 1 group 239.1.1.1 dev eth1 dstport 4789

The VXLAN interface is attached to the bridge: brctl addif br0 vxlan0 Ping tests initially fail until the VTEP learns the remote MAC. After the first successful ping, the bridge FDB shows a learned entry for the remote VTEP.

# bridge fdb show dev vxlan0
00:00:00:00:00:00 dst 239.1.1.1 via eth1 self permanent
2e:7a:a2:53:7b:31 dst 192.168.33.13 self

Packet captures illustrate ARP requests being multicast, replies being unicast, and subsequent ICMP traffic using the learned VTEP addresses.

Open vSwitch Implementation

OVS requires unicast VXLAN tunnels. The topology uses two OVS bridges: br‑int for logical network ports and br‑tun for VXLAN interfaces, connected by a PATCH port.

ovs-vsctl add-br br-int
ovs-vsctl add-br br-tun

ovs-vsctl add-port br-int patch-int -- set interface patch-int type=patch options:peer=patch-tun
ovs-vsctl add-port br-tun patch-tun -- set interface patch-tun type=patch options:peer=patch-int

Unicast VTEP ports are added for each remote host:

ovs-vsctl add-port br-tun vxlan0 -- set interface vxlan0 type=vxlan options:remote_ip=192.168.33.13
ovs-vsctl add-port br-tun vxlan1 -- set interface vxlan1 type=vxlan options:remote_ip=192.168.33.14

Flow tables are configured to separate traffic sources, handle broadcast traffic, and implement learning:

ovs-ofctl add-flow br-tun "table=0,priority=1,in_port=1 actions=resubmit(,1)"
ovs-ofctl add-flow br-tun "table=0,priority=1,in_port=5 actions=resubmit(,2)"
ovs-ofctl add-flow br-tun "table=0,priority=1,in_port=6 actions=resubmit(,2)"
ovs-ofctl add-flow br-tun "table=0,priority=0,actions=drop"

ovs-ofctl add-flow br-tun "table=1,priority=0,dl_dst=00:00:00:00:00:00/01:00:00:00:00:00 actions=resubmit(,20)"
ovs-ofctl add-flow br-tun "table=1,priority=0,dl_dst=01:00:00:00:00:00/01:00:00:00:00:00 actions=resubmit(,21)"

ovs-ofctl add-flow br-tun "table=21,priority=0,actions=output:5,output:6"

ovs-ofctl add-flow br-tun "table=2,priority=0,actions=learn(table=20,hard_timeout=300,priority=1,NXM_OF_ETH_DST[]=NXM_OF_ETH_SRC[],load:NXM_NX_TUN_ID[]->NXM_NX_TUN_ID[],output:NXM_OF_IN_PORT[]),output:1"

ovs-ofctl add-flow br-tun "table=20,priority=0,actions=resubmit(,21)"

After configuring the flows, pinging the remote network IP succeeds, and the flow table shows learned entries with priority 1 for the remote MAC.

# ovs-ofctl dump-flows br-tun table=20
cookie=0x0, duration=164.902s, table=20, n_packets=36, n_bytes=3360, hard_timeout=300, idle_age=19, hard_age=19, priority=1,dl_dst=ee:2c:09:42:0e:46 actions=load:0->NXM_NX_TUN_ID[],output:6

Further Reading

For detailed specifications, refer to the VXLAN RFC 7348.