How KeyDB Turns Redis into a Multi‑Threaded Database: Architecture Deep Dive

KeyDB is a fork of Redis that maintains 100% API compatibility while converting the originally single‑threaded key‑value store into a multi‑threaded architecture.

Thread Model

KeyDB splits Redis's original main thread into a main thread and multiple worker threads. Each worker thread is an I/O thread that listens on the port, accepts connections, reads data and parses the protocol.

KeyDB uses the SO_REUSEPORT feature, allowing multiple threads to bind to the same listening port.

Each worker thread is bound to a CPU core using SO_INCOMING_CPU, specifying which CPU receives the data.

After parsing the protocol, each thread operates on in‑memory data protected by a single global lock.

The main thread is also a worker thread (index 0 in the worker array) and performs tasks that only the main thread can handle, implemented in serverCron such as:

Statistics processing

Client connection management

DB resize and reshard

AOF handling

Replication synchronization

Cluster‑mode tasks

Connection Management

In Redis all connection management is performed by a single thread. In KeyDB each worker thread maintains its own set of connections; a connection is always created, used and destroyed within the same thread. A new field int iel; indicates the thread that owns the connection.

KeyDB maintains three key data structures for connection management:

clients_pending_write – per‑thread list of connections awaiting synchronous writes.

clients_pending_asyncwrite – per‑thread list of connections awaiting asynchronous writes.

clients_to_close – global list of connections that need to be closed asynchronously.

Separate synchronous and asynchronous queues are needed because some Redis APIs (e.g., pub/sub) involve publishing from one thread and delivering to a client handled by another thread.

Synchronous sending is performed entirely within the originating thread, as illustrated below:

Asynchronous sending involves two threads communicating via a pipe:

int fdCmdWrite; // write pipe
int fdCmdRead;  // read pipe

The local thread checks whether the client belongs to it; if not, it posts an AE_ASYNC_OP::CreateFileEvent to the owning thread, which then adds the write event, as shown:

Redis sometimes closes a client from a thread other than the one that owns the connection, so a global asynchronous close list is maintained:

Lock Mechanism

KeyDB implements a spin‑lock‑like mechanism called fastlock . Its main data structure includes:

int fdCmdWrite; // write pipe
int fdCmdRead;  // read pipe

Atomic operations (__atomic_load_2, __atomic_fetch_add, __atomic_compare_exchange) are used to test m_active == m_avail before acquiring the lock.

fastlock provides two acquisition methods:

try_lock – returns immediately on failure.

lock – busy‑waits, yielding the CPU after a large number of spins.

KeyDB combines try_lock with event handling to avoid busy‑waiting: each client has a dedicated lock; if lock acquisition fails, the operation is deferred to the next epoll_wait cycle.

Active‑Replica

KeyDB supports multi‑active replicas, allowing each replica to be writable. Key features include:

Each replica carries a UUID to break circular replication.

A new rreplay API packages incremental commands with the local UUID.

Keys and values carry a timestamp‑based version number; conflicts are resolved by comparing timestamps, using a scheme where the current timestamp is shifted left 20 bits and combined with a 44‑bit increment.

https://github.com/EQ-Alpha/KeyDB