ZestKV Achieves 2.3× SET Write Throughput, Surpassing pika3.5 at 700k QPS

Understanding the SET Write Path in ZestKV

To locate the performance bottleneck, the article first outlines the main path of a SET request, which consists of four stages linked by two queues: client → access (network read) → execution (lock + build write task) → persistence (batch write to storage engine) → response (assemble result + wake) → client.

Client
│
▼
Access (network read) → [Queue 1] → Execution (lock+build write task) → [Queue 2] → Persistence (batch write to storage engine) → [Response Queue] → Reply (assemble result + wake) → Client

The four stages have the following concurrency and responsibilities:

Access: multi‑lane, reads a full request from the connection and packages it as a task.

Execution: multi‑lane, locks the key, reads metadata, builds the write task and enqueues it.

Persistence: single‑lane before optimization, merges tasks into batches and writes to the WAL and memtable, then releases the lock and increments the version.

Response: single‑lane before optimization, takes tasks from the response queue, builds the RESP string, writes back to the connection and wakes the client.

Identifying the Bottleneck

High‑load testing shows that the front stages (access and execution) have ample capacity, while the back stages (persistence and response) are near full load and serialize, limiting overall throughput.

Persistence bottleneck: All writes funnel through a single serial path to the storage engine, preventing the engine’s native parallel memtable writes and causing idle CPU while waiting for log ACK.

Response bottleneck: Each command’s result calculation, RESP assembly, write‑back and wake‑up are performed on a single thread; as QPS rises, this stage saturates almost simultaneously with persistence.

Optimization 1: Parallelizing the Write Side

The single persistence lane is replaced by multiple queues, sharding writes by an ever‑increasing command sequence number. Commands that belong to the same original request share the same sequence, stay on the same lane, and are written together, preserving atomicity. Consistency is maintained because the record lock held from execution until persistence ensures that commands for the same key remain serialized, regardless of the sharding.

Network‑log ACK and memtable write are overlapped: the log is sent asynchronously while the memtable write proceeds; the request completes only when both finish, reducing the combined latency from 1 ms + 2 ms = 3 ms to max(1 ms, 2 ms) = 2 ms.

Optimization 2: Parallelizing the Response Side

After accelerating the write side, the response stage becomes the new limit. Two steps are applied:

Batch wake‑up: Instead of notifying the client per command, pending notifications for the same connection are merged and issued once per processing round. This eliminates a system call per command; SET QPS rises from 260 k to 300 k while preserving result order.

Connection‑based sharding: Responses are split across multiple lanes by connection, keeping all replies for a single connection on the same lane to preserve order. This avoids concurrency conflicts while allowing other connections to be processed in parallel.

The write side shards by command sequence number, while the response side shards by connection; the two dimensions are independent and together achieve full core utilization.

Consistency Guarantees Remain Intact

The redesign does not touch the “red lines”:

Durability: An OK is returned only after the log service acknowledges.

Write order: Preserved within each shard because the single‑lane write inside a shard is naturally ordered.

Command atomicity: Multiple tasks of one command share the same sequence and are written together.

Same‑key updates: Record lock ensures later commands wait for the previous command’s persistence before entering the queue.

Version control: After a successful write, the version is incremented, guaranteeing monotonic increase per key.

Performance Results

Benchmark numbers (all tests on identical hardware and load conditions):

pika 3.5: 350 k SET QPS (baseline).

ZestKV before optimization: 310 k SET QPS (0.89× baseline).

ZestKV after dual‑side async: 700 k SET QPS (2.06× pika 3.5, 2.32× before).

Write side changed from single‑lane to multi‑queue with command‑number sharding.

Response side changed from per‑command wake‑up to batch wake‑up + connection sharding.

Consistency and durability remain fully guaranteed.

The combined effect raises throughput from 260 k to 300 k via batch wake‑up, then adds write‑side parallelism and overlapping work to reach 700 k QPS.

Conclusion

The breakthrough rewrites ZestKV’s “one‑in, one‑out” serial pipeline into a dual‑side asynchronous engine: multi‑lane writes, overlapped network and CPU, batch wake‑ups and connection‑sharding on the response side. Parallelism is enabled by the existing record‑lock and version‑increment mechanisms, delivering a 2.3× self‑improvement and roughly double the performance of pika 3.5 without sacrificing any consistency or durability.