Building VXLAN Probes with evr: Monitoring Edge Routers Without Agents

evr is a network‑monitoring tool that measures packet loss, latency, and modification on the "Baidu network → EVR" link when neither the EVR device nor the opposite data‑center can host an agent.

Why evr is needed

EVR (Edge Virtual Router) is a VXLAN VTEP that connects a customer’s VXLAN overlay to Baidu’s backbone. Installing an agent on the EVR is prohibited, and placing a probe inside the customer’s rack is also impossible. The VTEP forwards the inner Ethernet frame according to the inner IP header after stripping outer headers, which enables a single‑sided probe.

VXLAN recap

VXLAN (RFC 7348) encapsulates a full Ethernet frame inside UDP, using a 24‑bit VNI to support ~16 million overlay networks. The packet layout from outer to inner consists of:

Outer Ethernet – 14 B – physical MAC – underlay L2 forwarding

Outer IP – 20 B – src/dst IP – underlay L3 routing (VTEP‑to‑VTEP)

Outer UDP – 8 B – dport = 4789 – VXLAN standard port

VXLAN header – 8 B – flags + VNI (24 bits) – identifies the overlay network

Inner Ethernet – 14 B – tenant MAC – start of the original L2 frame

Inner IP – 20 B – inner src/dst – real communication addresses inside the overlay

Inner UDP/TCP – 8 B – inner port – tenant traffic

Payload – N B – business data – original payload

The 8‑byte VXLAN header bit layout (the 5th bit "I" set to 1 marks the VNI as valid):

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|R|R|R|R|I|R|R|R|            Reserved                     |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                VXLAN Network Identifier (VNI) | Reserved |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

VTEP forwarding model

When a VTEP receives a VXLAN packet on UDP/4789, it strips the outer Ethernet/IP/UDP and the VXLAN header, then forwards the inner Ethernet frame according to the inner IP header. Setting the inner destination IP to the probe’s own address causes the VTEP to send the packet back to the probe.

evr design – three tricks

Trick 1: Self‑loop inner src & dst

Both inner source and destination IPs are set to the probe’s IP. The VTEP sees the inner dst = probe, reflects the packet, and the probe’s raw socket receives it without any server on the far side.

// agent.go: inner src == inner dst == local IP
inner, err := codec.EncodeVxlanInner(a.conf.VNI, a.conf.SrcMAC, a.conf.DstMAC,
    p.innerDstIP, p.innerDstIP, // inner src == inner dst == local IP
    newPort, a.conf.InnerDstPort,
    uint8(a.conf.TOS), a.conf.TTL, payload)

Trick 2: Embed target identifier in payload

Because inner src/dst are occupied, the EVR’s IP address is written into payload bytes 24‑28. When the reply arrives, the probe extracts these 4 bytes to map the packet back to the corresponding EVR.

// codec/packet.go: EVRCHECK protocol header
// offset 0  : magic "EVRCHECK" (8B)
// offset 8  : seq   (8B)  // atomic sequence number
// offset 16 : ts    (8B)  // send timestamp (ns)
// offset 24 : evrIP (4B)  // EVR source IP – used for mapping
// offset 28+ : salt  // padding for bit‑flip detection

Reply handling extracts the EVR IP:

// agent.go: handlePacket receives reply
seq, ts, evrIP := codec.DecodeWithSrcIP(payload)
 p := a.peerByEVRSrc[evrIP.String()]   // O(1) locate target

Trick 3: Spoof outer source IP (optional)

evr creates a raw socket with IP_HDRINCL so the kernel does not generate its own outer IP header. This allows a fake outer source IP ( mock_src) to be set, enabling quick failover without re‑configuring the EVR.

Full send‑receive flow

┌────────────────────────────┐   ┌────────────────────────────┐
│  Outer IP: mock_src → vtepIP│   │  Inner dst = local machine │
│  Outer UDP: srcPort → 4789 │   │  Payload contains EVR IP │
└────────────────────────────┘   └────────────────────────────┘
        │                               │
        ▼                               ▼
   VXLAN(VNI) ──► EVR strips outer headers
        │                               │
        ▼                               ▼
   Raw socket receives reflected packet
   → decode payload[24:28] → map to target EVR

Configuration

evr is driven by a JSON file. Each target entry consists of three fields: vtepIP (required) – the EVR VTEP’s IP (outer destination) evrSrcIP (required) – the identifier embedded in the payload for reply matching mockSrcIP (optional) – the outer source IP; if omitted, client_addr is used

{
  "id": "evr-probe-1",
  "client_addr": "203.0.113.10",
  "targets": "198.51.100.96#192.0.2.1#203.0.113.99,198.51.100.97#192.0.2.2#203.0.113.99",
  "vni": 15990000,
  "tos": 64,
  "ttl": 64,
  "client_port_range": "63000,63999",
  "rate_in_span": 2000,
  "span": "1s",
  "delay": "5s",
  "msg_len": 1024,
  "log_dir": "./log",
  "log_max_age_days": 3
}

Start evr with:

# Recommended: systemd as root
sudo ./evr -c /etc/evr/evr.json

# Quick CLI test
sudo ./evr \
  --client-addr 203.0.113.10 \
  --targets 198.51.100.96#192.0.2.1#203.0.113.99 \
  --rate-in-span 2000 --span 1s --msg-len 1024

# Override config fields with CLI flags
sudo ./evr -c /etc/evr/evr.json --rate-in-span 5000 --verbose

evr requires sudo (or CAP_NET_RAW) to create a raw ip4:udp socket, enable IP_HDRINCL, set IP TOS, and attach BPF. It runs only on Linux; macOS can compile but not execute the raw‑socket part.

Engineering tricks

Trick 1: BPF kernel‑level filtering

A raw socket receives all UDP packets, which can overwhelm a high‑throughput probe. A classic cBPF filter passes only packets that satisfy three conditions:

(1) IPv4 protocol = UDP (17)
(2) IPv4 TOS = cfg.tos
(3) UDP dst port = inner_dst_port (default 8972)

Only packets matching all three are delivered to user space, eliminating most overhead.

Trick 2: Source‑port rotation for ECMP coverage

Fixed five‑tuple traffic would always follow the same ECMP path. evr cycles the source port within client_port_range (e.g., 63000‑63999) so that successive probes explore the full ECMP hash space.

Trick 3: Four Salt patterns for bit‑flip detection

Bytes after offset 28 form a “salt” that rotates based on seq % 4: 0xFF – all 1 – detects any 1→0 flip. 0x00 – all 0 – detects any 0→1 flip. 0x5A – 01011010 – catches odd‑even errors on serial NIC links. 0xAAAA / 0x5555 – complementary – catches 1’s‑complement checksum blind spots.

If the received payload length matches the sent length, the probe compares the salt and logs a [client bitflip] entry when a mismatch occurs. This implementation is shared with the baize and kuiniu tools.

FAQ highlights

How does mock_src work? evr wraps the raw socket with ipv4.NewRawConn, which enables IP_HDRINCL. The kernel then transmits the manually crafted outer IPv4 header unchanged, preserving the fake source IP.

Is rate_in_span per‑target or total? It is the total rate across all targets. For 12 targets and rate_in_span=2000/s, each target receives roughly 166 pps.

Why set both inner src and dst to the probe IP? This makes the reflected inner frame’s destination the probe, allowing the raw socket to receive it without any server on the far side. Using the same IP for src also enables traceroute‑style TTL probing.

Why must evr run with sudo? Creating an ip4:udp raw socket, enabling IP_HDRINCL, setting TOS/DSCP, and attaching BPF all require CAP_NET_RAW, which is most easily granted via sudo or a systemd service running as root.

When to choose baize, kuiniu, or evr? Use baize for long‑term business‑network monitoring, kuiniu for GPU‑NIC (RoCE) paths in AI training, and evr for probing VXLAN/EVR paths where no server can be placed on the remote side.

Project repository: https://github.com/baidu/nettools