Unlocking 90 W/s Red Packet Peaks: Alibaba’s High‑Performance Local Storage Blueprint

Red Packet Business Characteristics

Alipay’s red‑packet service saw record participation in 2017, with a peak opening rate of 90 W/s on Chinese New Year’s Eve. The system’s architecture includes modular deployment, unified access, gateways, DAO, databases, and both online and offline data processing.

The database layer in the transaction chain has three key traits: massive peak pressure (up to millions of transactions per second during the draw), stringent latency requirements (sub‑100 µs), and built‑in disaster recovery via primary‑secondary replication.

Existing Block Storage Products

Alibaba offers three block‑storage options—SSD cloud disks, high‑efficiency cloud disks, and standard cloud disks. All provide 99.9999999 % reliability, but their IOPS and latency fall short of the red‑packet demands (target IOPS ≥ 200 k, latency < 100 µs).

High‑Performance Local Storage

To meet these requirements, a high‑performance local storage solution was designed, targeting ultra‑high IOPS and ultra‑low latency.

Key Components

Virtio‑blk : a semi‑virtualized block device interface that enables high‑speed data exchange between the VM and the physical host.

SPDK (Storage Performance Development Kit): a user‑space toolkit for building high‑performance, lock‑free storage applications.

NVMe SSD : PCI‑e SSDs using the NVMe protocol, offering high bandwidth and low latency.

Optimized Data Path

By retaining the standard file system and POSIX API, and streamlining the stack to only the block device layer, SPDK driver, and NVMe SSD, the data path is reduced from seven layers to four, dramatically cutting latency.

Performance Evaluation

Fio benchmarks on CentOS 7 show the high‑performance local disk achieving ~70 µs read latency and ~30 µs write latency, compared to ~130 µs and ~60 µs respectively for a generic virtualized setup.

Database tests reveal a jump from 14 242 TPS (RT 8.21 ms) to 26 969 TPS (RT 1.7 ms) when using the new storage.

Public Cloud Availability

Following its success in the Double‑11 and Chinese New Year red‑packet events, the solution—NVMe SSD + SPDK—was launched to Alibaba Cloud’s public services in February, becoming the world’s first cloud offering with this architecture.

Current specifications: 3 TB capacity, 500 k IOPS, 50 µs latency, 4 GB/s bandwidth.

Conclusion

Combining SPDK with NVMe SSD technology brings local disks close to physical‑machine performance, ensuring stable operation even under extreme peak loads such as Double‑11 and New Year red‑packet activities.