Introducing QDB and RebornDB: A Distributed Key‑Value Database Compatible with Redis and Supporting Dynamic Resharding

Why Choose a Key‑Value Database?

Many key‑value databases are used in real‑world systems, such as using Memcached to cache MySQL query results or MongoDB for document storage. Different scenarios require different key‑value solutions; Redis is a simple, fast, and versatile option, but it is limited by memory size.

QDB

QDB is a Redis‑compatible service that stores data on disk to overcome memory limits while keeping hot data in memory for performance.

Features

Redis Compatibility : Supports most Redis commands and data structures (String, Hash, List, Set, Sorted Set, etc.).

Disk Persistence : Hot data stays in memory; the rest is persisted to disk.

Multiple Backend Stores : Choose RocksDB, LevelDB, or GoLevelDB.

Bidirectional Sync with Redis : Can act as a Redis slave or master.

QDB uses LevelDB‑family storage engines built on LSM trees with Bloom filters and LRU caches for fast reads. RocksDB is preferred for production; GoLevelDB is suitable for quick experiments.

RebornDB

RebornDB is a proxy‑based distributed Redis cluster solution that improves on twemproxy by supporting dynamic topology changes.

Architecture

Clients connect to a reborn‑proxy , which parses RESP commands and forwards them to a backend store (reborn‑server or QDB). The system uses a Coordinator (Zookeeper or etcd) to manage metadata, a reborn‑config tool for group management, and a reborn‑agent for high availability.

Resharding

The data set is divided into 1024 slots; each slot is assigned to a group. Adding a new group involves migrating slots from existing groups. Example command:

reborn-config slot migrate 5 2 3

The migration uses a two‑phase commit to ensure safety, moving keys atomically from the source group to the target group.

High Availability

Reborn‑agent continuously monitors the health of applications (reborn‑proxy, backend stores, etc.). If a component fails, the agent restarts it or performs failover using the coordinator to promote a replica.

Future Work

Improve user experience by simplifying slot initialization, group management, and slot assignment.

Implement snapshot‑based copy migration to speed up large‑slot transfers.

Develop a polished dashboard and explore a P2P cluster architecture.

Conclusion

Building a distributed key‑value database is challenging, but RebornDB offers a Redis‑like interface with disk persistence, dynamic resharding, and high availability, making it a compelling choice for applications that need to store more data than Redis alone can handle.