How 25G SmartNICs Boost Cloud VM Networking Tenfold with Open vSwitch

Industry‑Leading SmartNIC Solutions Comparison

Traditional software virtual switches suffer performance bottlenecks because packets received from the physical NIC are handed to a vhost thread before reaching the VM, making vhost processing the limiting factor.

To offload traffic, many providers adopt 25G SmartNICs. Major implementations include AWS (ARM‑based many‑core), Azure (FPGA), Huawei Cloud (dedicated network processor), and Alibaba Cloud (programmable ASIC). Each has trade‑offs, with no single dominant solution.

Our Choice

We selected an open‑source Open vSwitch solution that leverages the Linux TC Flower offload interface because it minimally impacts existing control planes and can be rapidly deployed.

TC Flower classifies packets based on flow keys and applies actions such as drop, modify, or forward, similar to Open vSwitch’s match‑action model.

Open vSwitch‑Based SmartNIC Deployment

1. VM Migration

SmartNICs using VF passthrough are not migratable. Azure solves this with VF bonding and virtio‑net, but it requires user intervention. We adopted a VF + standby virtio‑net approach, allowing automatic bonding without manual configuration.

Create a virtio‑net NIC in the VM, attach a host VF with the same MAC address, and let the VM see two data‑plane interfaces.

The virtio‑net tap device is automatically added to the Open vSwitch bridge; when the VM switches NICs, the VF representor replaces the tap device.

2. VXLAN Encapsulation/Decapsulation Offload

Using Mellanox CX5, we found that the mlx5 driver did not support Ethernet‑over‑GRE offload, so we tested VXLAN. Although packets could be sent and received, offload to the NIC failed due to missing dport registration in newer kernels (e.g., u‑kernel 4.14).

We contributed a patch “datapath: support upstream ndo_udp_tunnel_add in net_device_ops” to resolve this.

3. Decapsulation Offload

After fixing the VXLAN issue, egress encapsulation offloaded, but ingress decapsulation still could not.

Analysis showed that the decap rule was applied to the VF port, while TC rules were set on the virtual device, leading to mismatched egress_dev flags. A patch to the mlx5 driver corrected the kernel compatibility check.

4. Backend Tap Device Encapsulation Drop

During live migration, Open vSwitch placed TC rules on the tap device, which forwarded packets to a gre_sys device that dropped them because the gre_sys device lacked a proper transmit function in the Open vSwitch kernel module.

We added logic to detect upstream tunnel support and adjusted the gre device handling accordingly.

5. Ethernet‑over‑GRE Offload

Even after applying Mellanox patches, ingress decapsulation could not offload because the gre_sys device did not create a TC ingress qdisc, preventing the driver from retrieving an ifindex for rule installation.

Upstream patches to make gre netdev types use TC rules and to disable checksum insertion for tunnel packets solved the issue.

Performance Comparison

After deployment, the solution achieved the following results:

Single VF receive performance: 15 Mpps.

Overall vSwitch forwarding performance (small packets): 24 Mpps.

Traditional software vSwitch: ~2 Mpps; virtual NIC receive: ~1.5 Mpps.

On an 8‑core VM handling 1‑byte UDP packets, PPS increased from 108 k to 4.69 M.

Future Plans

The solution is now live in public cloud as “Network Enhancement 2.0”, delivering over four times the network capability of the previous version. Upcoming work includes porting the design to bare‑metal cloud hosts for feature parity and exploring stateful firewall/NAT offload.