An Overview of SPDK (Storage Performance Development Kit) and Its Usage

SPDK (Storage Performance Development Kit) is an open‑source toolkit released by Intel in 2015 that connects Linux user‑space storage services directly to hardware, dramatically shortening the I/O path and exploiting lock‑free mechanisms to meet the bandwidth and latency demands of modern NVMe SSDs.

The kit’s motivation is twofold: running device drivers in user space (UIO) eliminates costly kernel context switches, and a polling‑based I/O model replaces interrupt‑driven processing, reducing latency and increasing throughput for flash‑based storage.

SPDK consists of three main sub‑components: a network front‑end (DPDK NIC driver and the user‑space network stack UNS), a processing framework that exposes an API for custom logic such as deduplication, compression, and encryption, and a back‑end layer offering drivers for NVMe, Linux AIO, RAMDISK, and Intel I/O acceleration devices.

The architecture is illustrated by a diagram showing the network front‑end, processing framework, and back‑end interacting to form a complete storage stack.

SPDK has been integrated into Ceph to overcome performance bottlenecks in flash environments, enabling Ceph to leverage SPDK’s high‑performance NVMe drivers and polling model.

At the block‑device layer (bdev), SPDK provides a pluggable API with drivers for NVMe, malloc (ramdisk), Linux AIO, virtio‑scsi, Ceph RBD, Pmem, and vhost‑SCSI, along with JSON‑RPC configuration, request queuing, timeout handling, and multiple lock‑free queues.

The driver layer includes a highly optimized, lock‑free NVMe driver and an IOAT copy‑offload engine that reduces DMA overhead for small I/O operations.

Above the block layer, the storage service layer offers a generic bdev abstraction and Blobstore, a lightweight, non‑POSIX file‑like interface designed for databases, containers, and virtual machines.

The storage protocol layer implements iSCSI, NVMe‑oF, and vhost‑SCSI targets, providing high‑throughput access to storage devices across different transport mechanisms.

To build and use SPDK, developers clone the repository from github.com/spdk/ , follow the installation guide, and run test commands such as:

# ll /sys/block/nvme*

#./perf_overlap -q 1 -s 8192 -w write -r 'trtype:PCIe traddr:0000:05:00.0' -d 2048MB -t 36000 -L -l -c 0x2 >> testlog.txt

SPDK’s implementation relies on the UIO driver interface, EAL for parsing sysfs nodes, and mmap to establish a data‑transfer channel between kernel and user space, exposing storage operations through a rich set of APIs.

An example C++ program demonstrates how to write a binary BLOB to a file and read it back using standard library streams:

#include <iostream>
#include <fstream>
#include <vector>

void WriteBlobToFile(const std::vector<char>& blobData, const std::string& filePath) {
    std::ofstream file(filePath, std::ios::binary);
    if (file.is_open()) {
        file.write(blobData.data(), blobData.size());
        file.close();
        std::cout << "BLOB data written to " << filePath << std::endl;
    } else {
        std::cerr << "Cannot open file: " << filePath << std::endl;
    }
}

std::vector<char> ReadBlobFromFile(const std::string& filePath) {
    std::ifstream file(filePath, std::ios::binary | std::ios::ate);
    if (file.is_open()) {
        size_t fileSize = file.tellg();
        file.seekg(0, std::ios::beg);
        std::vector<char> blobData(fileSize);
        if (file.read(blobData.data(), fileSize)) {
            file.close();
            return blobData;
        } else {
            file.close();
            throw std::runtime_error("Read error: " + filePath);
        }
    } else {
        throw std::runtime_error("Cannot open file: " + filePath);
    }
}

int main() {
    std::vector<char> blobData = {'H','e','l','l','o',',',' ','W','o','r','l','d'};
    WriteBlobToFile(blobData, "blob.txt");
    auto readData = ReadBlobFromFile("blob.txt");
    for (char c : readData) std::cout << c;
    return 0;
}

The article also discusses designing a Blob‑based file system on top of SPDK, including threading, JSON configuration parsing, a four‑layer architecture (physical, logical, control, service), Git version management, radix‑tree memory management, syscall hooking, and POSIX API compatibility considerations.