How KeyDB Transforms Redis with Multithreading: Architecture Deep Dive

KeyDB is a fork of Redis that adds multithreading while remaining 100% compatible with the Redis API.

Project address: https://github.com/JohnSully/KeyDB

Multithreaded Architecture

Thread Model

KeyDB splits Redis's original single main thread into a main thread and multiple worker threads. Each worker thread is an I/O thread that listens on the port, accepts connections, reads data, and parses the protocol.

KeyDB uses the SO_REUSEPORT option so multiple threads can bind to the same listening port.

Each worker thread also binds to a specific CPU using the SO_INCOMING_CPU feature, designating which CPU receives the data.

After parsing the protocol, each thread operates on in‑memory data protected by a single global lock.

The main thread is actually worker thread 0 and performs the same work as other workers plus tasks only the main thread can do, such as serverCron.

Statistics processing

Client connection management

Database resize and reshard

AOF handling

Replication master‑slave synchronization

Cluster‑mode tasks

Connection Management

In Redis all connection management runs in one thread. KeyDB assigns each worker thread a set of connections, maintaining them in a thread‑local list; creation, operation, and destruction of a connection must stay in the same thread.

int iel; /* the event loop index we’re registered with */

This field indicates which thread handles the connection.

KeyDB maintains three key data structures for connection management: clients_pending_write: thread‑local list of connections waiting to send data synchronously. clients_pending_asyncwrite: thread‑local list of connections waiting to send data asynchronously. clients_to_close: global list of connections that need to be closed asynchronously.

Separating synchronous and asynchronous queues handles cases like Pub/Sub where the publishing thread differs from the subscriber’s thread.

When a local thread needs to send data asynchronously, it checks whether the client belongs to the local thread; if not, it obtains the client’s owning thread ID and enqueues a write event via AE_ASYNC_OP::CreateFileEvent. The owning thread processes the pipe message and adds the request to its write events.

Some client‑close requests are not executed in the connection’s thread, so a global asynchronous close list is maintained.

Lock Mechanism

KeyDB implements a spin‑lock‑like mechanism called fastlock.

struct ticket {
    uint16_t m_active; // unlock +1
    uint16_t m_avail;  // lock   +1
};

struct fastlock {
    volatile struct ticket m_ticket;
    volatile int m_pidOwner; // thread ID that currently holds the lock
    volatile int m_depth;    // recursion depth for the owning thread
};

Atomic operations such as __atomic_load_2, __atomic_fetch_add, and __atomic_compare_exchange are used to compare m_active and m_avail to decide if the lock can be acquired.

fastlock provides two lock acquisition methods: try_lock: attempts once and returns immediately on failure. lock: busy‑waits; after 1,048,576 attempts it calls sched_yield to yield the CPU.

KeyDB combines try_lock with the event loop to avoid busy‑waiting. Each client has a dedicated lock; before reading client data, the lock is tried. If it fails, the operation is skipped and will be retried in the next epoll_wait cycle.

Active‑Replica

KeyDB implements an active‑replica mechanism where each replica can be writable (non‑read‑only) and synchronizes data with other replicas. Key features include:

Each replica has a UUID to eliminate circular replication.

A new rreplay API packages incremental commands with the local UUID.

Keys and values carry a timestamp‑based version number; if a local key has a newer timestamp than the incoming one, the write is rejected. The version is generated by left‑shifting the current timestamp by 20 bits and adding a 44‑bit increment.