How ZestKV Solves Pika’s Limits with a Serverless KV Architecture

Background

Pika is a Redis‑compatible key‑value store built on RocksDB for disk‑based storage. It solves Redis’s memory capacity limitation and cost issues and was open‑sourced in 2016 (≈6k Stars). The design suffers from high horizontal scaling cost, local‑disk capacity limits, operational overhead, and data‑loss risk due to asynchronous binlog replication.

ZestKV Overview

ZestKV is a cloud‑native, serverless KV service launched in 2024 to address Pika’s shortcomings. It separates storage and compute, provides elastic scaling and multi‑tenant isolation.

Cluster Architecture

ZestManager : Central management node that stores metadata in ETCD and coordinates ordered cluster operations.

ZestPika : Stateless compute node that processes client read/write requests based on shard assignments from the manager.

ZestProxy : Stateless proxy that routes client requests to the appropriate compute node using up‑to‑date routing tables.

ZestSstServer : Consumes data from the log service, materializes SST files, uploads them to S3, and caches files locally to reduce S3 I/O.

Log Service : Independent high‑availability logging service.

S3 : Object storage for persisted RocksDB‑Cloud engine files.

Compact‑Runner : Stateless nodes that execute compaction tasks.

Core Features

Storage‑Compute Separation

ZestKV adopts a shared‑storage model where the storage engine (RocksDB‑Cloud) persists data to S3 while compute nodes remain stateless. Write flow: client → proxy → compute node → log service → RocksDB‑Cloud MemTable → immutable MemTable → SST generation → S3 upload. Each SST and MANIFEST file carries an epochId generated at instance start; leader changes create a new epoch, preventing object‑key conflicts in S3.

Recovery uses a replication_sequence stored in the MANIFEST. After a restart, a node reads the latest replication_sequence from S3, resumes consumption from the log service at the correct offset, and reconstructs the MemTable.

Multi‑Tenant Support

Each tenant receives an isolated Database with its own shard set and routing strategy. The manager creates a Database, assigns a unique ID, provisions default region shards, and distributes routing tables to proxies and compute nodes, ensuring data and resource isolation.

Elastic Scaling

Data is partitioned into range‑based Regions (similar to HBase). Regions can be split, merged, or migrated:

Split/Merge : The manager initiates a split, creates offline child Regions, materializes new RocksDB‑Cloud instances, and updates routing tables. Ingestion is performed via a modified RocksDB‑Cloud ingest interface that injects existing SST files without copying them. Alias metadata maps S3 objects to logical file names, and local hard‑links reduce cold‑read latency.

Migration : A three‑phase process—(1) manager creates a migration task and notifies source/target nodes via ETCD; (2) source Region is set to read‑only (writes return retry); (3) manager updates the routing table and propagates it to proxies and compute nodes. The whole migration completes in seconds with only a brief write pause.

Automatic Scaling on Kubernetes

A dedicated serverless‑manager pod periodically fetches metrics via the K8s OpenAPI, decides whether to increase or decrease pod replicas, and triggers shard rebalancing and routing updates. This enables per‑Database resource isolation and scaling based on load and data volume.

Performance Evaluation

Using redis‑benchmark, ZestKV Serverless was compared with Pika 3.5 (single‑group and dual‑group deployments). In read workloads, ZestKV matched Pika’s QPS and latency. In write workloads, ZestKV achieved ~40 % higher throughput thanks to asynchronous batch processing and SST generation.

Future Directions

Support for multi‑model databases (e.g., tabular, time‑series).

Fine‑grained multi‑tenant resource isolation via resource groups and proactive rate limiting.