High‑Performance Storage with SPDK vhost in Didi Cloud Enhanced DC2

SPDK (Storage Performance Development Kit), initiated by Intel, is a user‑space, asynchronous, lock‑free, polling‑based NVMe driver library that dramatically reduces NVMe command latency and increases per‑CPU IOPS compared to the traditional Linux kernel NVMe driver.

In Didi Cloud's high‑performance solution Enhanced DC2, the SPDK vhost framework is customized to provide high‑throughput I/O for virtual machines, as illustrated by the following architecture diagram:

SPDK vhost runs as a user‑space process on the host, polling a shared ring buffer to receive VM I/O requests, processing them without locks or interrupts, and notifying the VM via the same ring buffer.

VM storage front‑end driver (e.g., virtio‑blk, virtio‑scsi, or native NVMe driver) receives I/O and places it into the ring buffer.

Shared ring buffer enables zero‑copy communication between the VM front‑end and the host back‑end.

SPDK vhost back‑end implements the actual I/O processing logic in user space.

Traditional VM I/O paths involve long kernel call stacks, causing high latency and reduced throughput. By using SPDK + vhost, the I/O path is shortened, achieving low latency and high IOPS, as shown in the following diagrams.

The VM storage front‑end can use virtio‑blk for fast data transfer or a native NVMe driver to interact directly with the SPDK vhost back‑end, eliminating protocol conversion overhead.

SPDK (Storage Performance Development Kit) is an Intel‑initiated toolkit for building high‑performance, scalable user‑space storage applications, reusing many DPDK components such as memory management and poll‑mode drivers. Its core principles are lock‑free operation and poll‑mode processing.

Key SPDK design points:

NVMe protocol implementation resides in user space, allowing applications to access devices via SPDK APIs without kernel drivers.

Lock‑free architecture avoids contention overhead.

Polling mode eliminates interrupt‑related context switches.

Enhanced DC2 employs Intel high‑performance NVMe SSDs, leveraging the NVMe protocol’s superior bandwidth and lower latency over traditional SCSI.

To productize SPDK, hot‑upgrade support for vhost and QEMU, as well as automatic recovery for unexpected vhost exits, are integrated, enabling seamless online fixes without VM disruption.

Performance testing with fio (IO scheduler=noop) used the following parameters for IOPS measurement:

--bs=4k

--rw=randread/randwrite

--direct=1

-numjobs=4

-iodepth=128

Results: random read IOPS ≈ 730,000 and random write IOPS ≈ 470,000.

Latency testing with fio (engine=psync) used:

--ioengine=psync

--bs=4k

--rw=randread/randwrite

--direct=1

-numjobs=1

-iodepth=1

Latency results: random read ≈ 96 µs, random write ≈ 23 µs.

Beyond storage, Enhanced DC2 provides strong compute (Intel Xeon Platinum 8160 with AVX‑512) and networking (DPDK‑based high‑bandwidth, low‑latency) capabilities.

Disclaimer: Please credit the original author and source when reposting.