Accelerating Virtual Machine Storage with SPDK vhost and NVMe SSDs

SPDK (Storage Performance Development Kit) is an Intel‑initiated user‑space library that accelerates applications using NVMe SSDs as backend storage. Its core provides a user‑mode, asynchronous, lock‑free, polling‑based NVMe driver.

Compared with the traditional Linux kernel NVMe driver, SPDK dramatically reduces NVMe command latency and increases per‑CPU IOPS, enabling cost‑effective solutions such as SPDK‑based vhost for cloud VM storage acceleration.

The enhanced DC2 solution in Didi Cloud improves performance by extending the vhost framework in SPDK. The vhost architecture consists of three main parts:

Virtual machine storage front‑end driver (e.g., virtio‑blk, virtio‑scsi, or native NVMe driver) that receives I/O requests and places them into a shared Ring Buffer.

Shared Ring Buffer – a memory region shared between the VM front‑end and the host storage back‑end for transferring commands and results.

SPDK vhost back‑end – the user‑space storage back‑end that processes I/O requests with lock‑free, poll‑based logic.

In traditional VM I/O paths, requests traverse a long kernel call stack on both the guest and host, causing high latency and reduced throughput. By using SPDK + vhost, the I/O path is shortened, achieving low latency and high IOPS.

Two front‑end options are discussed:

virtio‑blk provides a semi‑virtualized block interface that optimizes data transfer between guest and host.

Native NVMe driver can communicate directly with the SPDK vhost back‑end, eliminating protocol conversion overhead for even higher performance.

Key SPDK features highlighted include:

User‑space implementation of the NVMe protocol, allowing applications to access devices via SPDK APIs without kernel drivers.

Lock‑free design to avoid contention overhead in multi‑threaded environments.

Polling mode to eliminate interrupt‑driven context switches.

The solution leverages high‑performance Intel NVMe SSDs, which offer higher bandwidth and lower latency than traditional SCSI.

Productization adds vhost hot‑upgrade, QEMU hot‑upgrade, and automatic recovery for unexpected vhost exits, enabling seamless online upgrades with minimal impact on guest workloads.

Performance testing with fio (IO depth 128, 4 KB random read/write, noop scheduler) shows the enhanced DC2 achieving approximately 730 k IOPS for random reads and 470 k IOPS for random writes. Latency tests (single‑job, depth 1) report ~96 µs read latency and ~23 µs write latency.

Hardware specifications of the enhanced DC2 include Intel Xeon Platinum 8160 CPUs with AVX‑512 support and DPDK‑based networking for high bandwidth and low latency.