How ZestKV’s Compute‑Storage Separation Cuts Costs, Boosts Stability, and Scales Seamlessly

ZestKV is a Redis‑compatible distributed KV store that separates compute and storage, enabling lower cost, higher stability, and elastic scaling.

Architecture Overview

Three‑layer service model built on a log service and object storage.

Proxy layer : entry point, receives requests and forwards them to the appropriate compute node.

Compute layer : stateless engine that parses Redis protocol, executes commands, maintains a local cache; does not store data permanently.

Control layer : “brain” that manages data distribution, routing, orchestrates scaling, migration, and health checks.

Storage layer : persists data to files, performs compaction, serves read requests, and synchronizes files to object storage.

Log service : reliably records every write as an immutable log, guaranteeing no data loss.

Object storage : low‑cost, highly reliable long‑term repository for the final data files.

Operation Flow

Write Path

flowchart LR
A["Business write"] --> B["Compute node executes command"]
B --> C["Write to log service (record change)"]
C --> D["Immediately return success"]
C -. Asynchronous .-> E["Storage layer consumes log and materializes data"]

Writes are synchronous to the client but asynchronous to the storage; once the log records the change, the client receives success, ensuring low latency while the storage layer later materializes the data.

Read Path

flowchart LR
A["Business read"] --> B["Compute node checks local cache"]
B -- hit --> C["Return directly (fastest)"]
B -- miss --> D["Request storage layer"]
D -- storage miss --> E["Fetch from object storage"]

Reads follow a cache‑first strategy; cache hit returns instantly, otherwise the request is forwarded to the storage layer, which may pull data from object storage if not locally cached.

Design Rationale

Traditional Redis/Pika couples compute and storage, leading to three problems when scaling: expensive and slow expansion, low resource utilization, and costly multi‑replica storage. ZestKV addresses these by:

Compute‑storage separation : independent scaling of compute and storage.

Log‑as‑source of truth : all writes first land in a reliable log, then are uniformly materialized by the storage layer, ensuring a consistent global view.

Uniform materialization avoids inconsistencies that would arise if each compute node performed its own compaction.

Business Benefits

Cost reduction : data resides in cheap object storage; compute nodes are lightweight, lowering overall hardware expense.

Zero migration effort : full Redis protocol compatibility lets existing applications switch endpoints without code changes.

High reliability : the log service guarantees that a successful write is never lost; object storage provides durable long‑term retention; stateless compute nodes can be replaced without data loss.

Scalable growth : compute or storage can be expanded independently based on workload, eliminating full‑cluster re‑sharding.

Glossary (Key Terms)

Redis protocol : widely used storage access protocol; compatibility means no application changes.

Compute‑storage separation : architectural pattern that decouples processing from data persistence.

Compute node : stateless service handling read/write requests and caching.

Storage layer : persists data, manages files, and serves reads.

Log service : immutable, highly reliable write‑ahead log.

Object storage : massive, low‑cost, highly reliable storage backend.

Materialization : converting logged writes into queryable data files.

Cache : fast‑access layer for hot data to accelerate reads.