ZestKV: 360's Cloud-Native Serverless KV Store with Compute-Storage Separation

Birth of ZestKV

Redis dominates as a high-performance KV solution with a mature ecosystem, but as data grows from tens of GB to hundreds of GB or TB, memory costs, capacity planning, cluster scaling, and peak resource reservation become major cost drivers. Pika addresses capacity and cost by using RocksDB with local disks, yet its traditional local-storage architecture binds compute and data: deployment environment dictates compute capacity; scaling out requires not only adding nodes but also migrating historical data, incurring extra network I/O and manual operational overhead.

ZestKV's core value lies in decoupling "data capacity" from "compute capability": all persistent data resides in shared object storage, real-time incremental writes go to a highly available log service, and compute nodes primarily serve user requests. This allows data capacity to grow independently while compute resources scale closer to actual QPS and business fluctuations.

ZestKV is a self-developed Serverless KV storage service by 360 Zhihui Cloud , compatible with the Redis protocol. It re-architects compute, log, object storage, background LSM tasks, and metadata management into separate components, making compute nodes more stateless so that scaling relies on region metadata and routing changes rather than moving massive historical data.

Core Architecture: Decoupled Compute, Log, Storage, and Background Tasks

The architecture comprises the following components:

ZestProxy : Redis request entry and routing; forwards requests to compute nodes based on region routing. Clients continue using standard Redis access patterns.

ZestPika / Compute Node : Handles GET/SET and other read/write requests; compute nodes trend toward statelessness, allowing compute resources to scale independently.

Log Service : Stores real-time incremental writes / WAL; provides incremental data needed for real-time persistence and failure recovery.

ZestSstserver : Consumes logs, materializes SST files, uploads them to S3, and handles some background tasks; reduces impact of flush and other background work on foreground requests.

S3 / Object Storage : Holds RocksDB-Cloud full data and metadata; breaks local disk capacity boundaries.

ZestManager + ETCD : Maintains metadata, regions, routing, and cluster scheduling; centrally drives splitting, migration, and routing changes.

Compact Runner : Executes compaction independently; isolates LSM tree background pressure from online compute nodes.

Key architectural shift : Traditional Pika follows "where data is, compute is there"; ZestKV hands real-time increments to the log service and full data to object storage, while compute nodes focus on request processing. For operations, the scaling target shifts from "stateful data nodes" to "elastic compute resources".

Write path : Client → ZestProxy → Compute Node. Incremental writes first enter the highly available Log Service, then update the RocksDB-Cloud MemTable. Sstserver consumes logs, materializes SSTs, and persists them to S3. Background tasks like compaction are offloaded to separate components, reducing resource contention on online compute nodes.

Seven Core Capabilities: From "Can Store" to "Elastic, Stable, Scalable"

1. Compute-Storage Separation: Large Capacity No Longer Means Large Compute Specs

In traditional Redis/Pika architectures, data scale often directly dictates instance specs. Even with low average QPS, large data volumes force long-term retention of corresponding memory or local disk resources. ZestKV places full data in object storage and real-time increments in a highly available log service, while compute nodes provide capacity based on request load. Thus "how much data to store" and "how much compute is needed" can be planned separately.

Typical benefit scenario : A business with 500 GB data, average 5,000 QPS, and peak 100,000 QPS no longer needs to maintain peak-level compute resources just because of 500 GB data. For large-capacity, low average QPS, or tidal traffic patterns, compute-storage separation improves resource utilization.

2. Second-Level Region Splitting and Migration: Scaling Avoids Physical Data Movement

ZestKV manages data via Range/Region. During region splitting, it reuses SST files in object storage through RocksDB-Cloud Ingest and metadata mechanisms, avoiding full replication of historical engine files to new nodes. Scaling thus shifts from "moving data" to "modifying MANIFEST, region metadata, and routing", greatly weakening the relationship between scaling time and existing data volume. Region migration is similarly driven by the central node through state transitions, brief write pauses, and routing switches.

3. Serverless Elasticity: Beyond Kubernetes HPA

The challenge of database serverless is not merely spinning up more pods, but handling sharding and routing after new compute resources arrive. ZestKV's serverless-manager combines CPU, read/write QPS, and region data volume metrics, coordinating Kubernetes, Manager, region split/merge, and migration to achieve database-level elasticity.

Initial workloads can run on shared compute resources.

As QPS or data volume grows, regions migrate to dedicated pods.

Further growth triggers region splitting to increase parallel processing.

During traffic drop-off, regions merge and migrate back to shared resources, releasing idle compute.

4. Background Task Decoupling: Reducing Compaction Interference on Online Requests

RocksDB-style databases under high write loads often suffer from flush, L0 accumulation, and compaction impacting CPU, disk I/O, and P99 latency. ZestKV splits SST materialization, flush, compaction, and other background work into independent components, letting online compute nodes focus on user requests. For DBAs/SREs, this design not only raises peak QPS but more importantly lowers the probability of background LSM tasks contending with business threads, architecturally improving performance stability under high load.

5. Write Path Dual-Side Asynchronization: Unlocking Multi-Core Parallelism

ZestKV's performance optimization targets the main-path serial bottleneck: the network side upgrades from a single task queue to multiple lock-free queues; the write side evolves from single-path submission to multi-path parallelism; the response side uses batched wake-ups and per-connection sharding. In internal benchmarks, SET QPS rose from 310K before optimization to 700K after; under the same conditions Pika 3.5 achieved 350K.

6. Data Consistency: Safeguarding Correctness During Writes and Migration

ZestKV ensures data consistency during concurrent writes through log persistence, per-key serial processing, and version control. During region splitting and migration, the Manager centrally coordinates state synchronization, brief write pauses, and routing switches, guaranteeing data integrity and supporting rollback on anomalies.

7. Redis Protocol Compatibility: Lowering Migration and Learning Costs

ZestKV continues to use the Redis protocol and client access patterns, so developers need not learn a new database SDK. Existing Redis/Pika workloads can undergo compatibility verification and gradual migration while preserving their primary data access models.

Note : ZestKV is a distributed KV; not all Redis semantics map directly. Production migration must confirm compatibility scope against the actual command set used.

Performance Data: R&D Benchmarks Show Upper Limits, DBA Tests Show Reality

1. R&D Benchmark: SET Dual-Side Async Optimization

R&D benchmark data illustrates write-path optimization effects; avoid equating it directly with all production workloads.

Pika 3.5 : 350K SET QPS (baseline)

ZestKV before optimization : 310K SET QPS (0.89× Pika 3.5)

ZestKV after dual-side async optimization : 700K SET QPS (~2.06× Pika 3.5 / ~2.32× before optimization)

2. DBA Real-World Stress Test: Throughput Approaches Redis/Pika

DBA stress tests covered 6 scenarios. Aggregate results:

Pika Replica : Average best QPS ~319,636; median best P99 ~18.17 ms

ZestKV : Average best QPS ~279,089; median best P99 ~27.03 ms

Redis : Average best QPS ~274,096; median best P99 ~18.71 ms

Interpretation : Under the current test model, ZestKV's throughput reaches near Redis/Pika levels, but P99 latency remains higher. This indicates ZestKV's primary advantages should focus on "large capacity + elasticity + compute-storage separation + resource utilization" rather than beating in-memory Redis on latency for all scenarios.

3. Single-Item Peak Reference

Peak QPS numbers vary by client connection count (c) and payload size; the table below shows each system's peak capability under its own optimal connection count, not a strict apples-to-apples comparison.

Pika Replica : GET max 600,154 (128B/P10/c=512); SET max 242,131 (128B/P10/c=512)

ZestKV : GET max 596,199 (128B/P10/c=4000); SET max 236,423 (128B/P10/c=2000)

Redis : GET max 587,100 (128B/P10/c=512); SET max 253,536 (128B/P10/c=512)

Redis / Pika / ZestKV Selection Guide

1. Boundary Comparison

Core Positioning : Redis – ultra-low latency, pure hot data, in-memory KV; Pika – large capacity, Redis-compatible, disk KV; ZestKV – large capacity, elasticity, cloud-native serverless KV.

Data Medium : Redis – memory; Pika – local SSD/RocksDB; ZestKV – object storage + log service.

Compute-Storage Coupling : Redis – tightly bound; Pika – bound to local disk; ZestKV – decoupled.

Horizontal Scaling : Redis – cluster data redistribution; Pika – with data migration; ZestKV – region split/migration, no physical data movement.

Elasticity & Multi-Tenancy : Redis – fixed instances/data isolation; Pika – fixed nodes/data isolation; ZestKV – K8S auto-scaling, dual isolation of data and resources.

Typical Strengths : Redis – low latency, mature ecosystem; Pika – large capacity, low cost; ZestKV – large capacity + elasticity + stability + resource utilization.

Selection Principle : Small data, all hot, extreme low-latency needs → Redis. Large capacity but fixed nodes acceptable → Pika. If business shows continuous capacity growth, frequent scaling, compaction jitter, pronounced traffic peaks/valleys, or low resource utilization → ZestKV warrants PoC and migration evaluation.

2. Compatibility Boundaries: Redis Usage Compatibility ≠ Full Compatibility

ZestKV maintains compatibility with the Redis protocol and mainstream access patterns, but its distributed architecture imposes limitations on commands relying on global state, cross-shard atomicity, or single-instance topology. Technical advocacy should proactively disclose boundaries to avoid post-launch semantic surprises.

Four command categories needing attention :

Cluster/Topology Management : BGSAVE, SHUTDOWN, SLAVEOF, CONFIG, SLOWLOG, etc.

Global Scan & Global State : KEYS/SCAN cannot provide Redis single-instance style global scan; INFO, DBSIZE, etc.

Cross-Shard Multi-Key / Transaction : MULTI/EXEC/WATCH, MSETNX, BITOP, RPOPLPUSH, SMOVE, SUNION, SINTER, etc.

Pub/Sub & Module Extensions : Pub/Sub cross-node propagation, some Redis Modules or complex Lua scripts require verification against current command matrix.

Known semantic difference: HSET/HMSET . Redis returns "number of newly added fields"; ZestKV currently returns a constant 1. Code relying on return values for business logic must be refactored before migration.

Correct pre-migration practice : Do not substitute "Redis compatible" for command auditing. Extract real command sets from production call chains, slowlog/command statistics, SDK wrappers, or test traffic, and build a compatibility matrix classifying each command as "supported", "semantic difference", "cross-shard limited", or "unsupported".

3. Developer and Operations Recommendations

Use multiple TCP connections on the client side : ZestKV server uses multi-threading, but requests on a single TCP connection are serialized. High-concurrency workloads should establish multiple connections via a connection pool for parallel request sending.

Avoid hot keys and monitor key prefix distribution : Same key is serialized server-side; ZestKV defaults to using the first 8 bytes of the key as sharding basis.

Design retries for failures during split/migration : Include exponential backoff, max retry count, and failure logging.

Use DBREGIONS + REGIONSCAN for full key traversal : In distributed environments, KEYS/SCAN cannot provide single-instance style global scan.

Best-Fit Scenarios for ZestKV

Redis/Pika workloads with continuously growing data : Data growing from 100 GB, 300 GB to TB level, where scaling becomes routine.

Large capacity, low average QPS : User profiles, historical states, product/content data, IoT device states, long-cycle KV.

Pronounced peak/valley or burst traffic : Big promotions, game launches, event systems, online education, scheduled reporting.

Multi-business shared KV platform : Multiple business lines sharing infrastructure, with routing and resource isolation per database.

Online businesses sensitive to RocksDB background jitter : Already experiencing CPU, I/O, or P99 spikes due to flush/compaction.

Scenarios not recommended for priority migration : ZestKV does not aim to replace all Redis. Small data, purely hot data, and extreme single-request low-latency needs still favor in-memory Redis. Workloads heavily dependent on currently uncovered special commands or Redis single-instance global semantics should complete compatibility refactoring before considering migration.

Recommended Deployment Approach: PoC First, Then Gradual Migration

For existing Redis/Pika workloads, avoid deciding migration based on a single benchmark. A safer path is to pick a representative workload and validate with real command sets, real value sizes, and real connection models.

Catalog real command sets : Collect Redis/Pika commands, pipelines, transactions, Lua, TTL, big keys, connection counts, and hot keys.

Import real data samples : Verify key distribution, value sizes, hot key ratios, and capacity growth models.

Run performance comparisons : Focus on QPS, P50/P99, CPU, network, error rates, and latency during scaling events.

Validate failure scenarios : Cover compute node restarts, region split/migration, network jitter, high load, and component failures.

Gradual migration : Prioritize large-capacity, cost-sensitive, peak-valley, or frequently scaling workloads; expand scope incrementally.

Signals that a PoC is warranted : If the business already faces high Redis memory costs, continuous Pika data growth, increasingly frequent scaling, noticeable compaction impact on online latency, or large traffic peak-valley differences, ZestKV deserves the next round of architectural evaluation and real-workload stress testing.