How KeyDB Transforms Redis into a Multi‑Threaded Database

KeyDB is a fork of Redis that maintains 100% compatibility with the Redis API while converting Redis into a multi‑threaded system.

Thread Model

KeyDB splits Redis's original main thread into a main thread and multiple worker threads. Each worker thread is an I/O thread responsible for listening on ports, accepting connections, reading data, and parsing the protocol.

KeyDB uses the SO_REUSEPORT feature, allowing multiple threads to bind to the same listening port.

Each worker thread also binds to a specific CPU using the SO_INCOMING_CPU feature to designate which CPU receives data.

After parsing the protocol, each thread operates on in‑memory data protected by a global lock to control concurrent access.

The main thread is itself a worker thread (index 0 in the worker array) and additionally performs tasks that only the main thread can execute.

Main thread responsibilities (implemented in serverCron) include:

Processing statistics

Client connection management

Resizing and reshaping DB data

Handling AOF

Replication master‑slave synchronization

Cluster‑mode tasks

Connection Management

In Redis, all connection management is handled by a single thread. In KeyDB, each worker thread manages its own set of connections, inserting them into a thread‑local connection list; creation, operation, and destruction of a connection must occur within the same thread.

int iel; /* the event loop index we're registered with */

KeyDB maintains three key data structures for connection management: clients_pending_write: thread‑local list for synchronously sending data to client connections clients_pending_asyncwrite: thread‑local list for asynchronously sending data to client connections clients_to_close: global list for connections that need to be closed asynchronously

Separating synchronous and asynchronous queues handles cases where a publishing thread differs from the subscriber's thread, ensuring correct delivery of messages.

When a local thread needs to send data asynchronously, it checks whether the client belongs to the local thread; if not, it obtains the client’s owning thread ID and enqueues a write event via AE_ASYNC_OP::CreateFileEvent. The owning thread then processes the pipe message and adds the request to its write events.

int fdCmdWrite; // write pipe
int fdCmdRead;  // read pipe

Some client‑close requests are not executed in the thread that owns the connection, so KeyDB maintains a global asynchronous close list.

Lock Mechanism

KeyDB implements a spin‑lock‑like mechanism called fastlock.

Key data structures of fastlock:

struct ticket {
    uint16_t m_active; // unlock +1
    uint16_t m_avail;  // lock +1
};
struct fastlock {
    volatile struct ticket m_ticket;
    volatile int m_pidOwner; // thread ID that currently holds the lock
    volatile int m_depth;    // recursion depth for the owning thread
};

Atomic operations such as __atomic_load_2, __atomic_fetch_add, and __atomic_compare_exchange compare m_active and m_avail to determine lock acquisition. fastlock provides two acquisition methods: try_lock: returns immediately if the lock cannot be obtained lock: busy‑waits, performing up to 1024 × 1024 iterations before yielding the CPU with sched_yield KeyDB combines try_lock with the event loop to avoid busy‑waiting: each client has a dedicated lock; before reading client data, the lock is attempted, and if it fails, the operation is deferred to the next epoll_wait cycle.

Active‑Replica

KeyDB implements an active‑replica mechanism where each replica can be writable (non‑read‑only) and synchronizes data with other replicas. Key features include:

Each replica has a UUID to eliminate circular replication

A new rreplay API packages incremental commands with the local UUID

Keys and values carry a timestamp version; writes with older timestamps are rejected, using a timestamp composed of the current time shifted left 20 bits plus a 44‑bit increment

Project Address

https://github.com/JohnSully/KeyDB