KeyDB Multithreaded Architecture and Design Overview

KeyDB is a high‑performance, multithreaded fork of Redis that retains full compatibility with the Redis protocol, modules, and scripting while delivering up to twice the query throughput and 60 % lower latency on the same hardware.

The project’s source is available at https://github.com/JohnSully/KeyDB , and the article is based on a source‑code review.

Multithreaded Architecture

Thread Model

KeyDB splits Redis’s original single main thread into one main thread plus multiple worker I/O threads. Each worker listens on the same port (SO_REUSEPORT), is bound to a CPU core, reads and parses client requests, and then operates on shared memory protected by a global lock. The main thread is also a worker (index 0) and runs the serverCron tasks such as statistics, client connection management, database resizing, AOF handling, replication, and cluster duties.

Connection Management

In KeyDB each client connection carries an int iel; /* the event loop index we're registered with */ indicating which worker thread owns it. Three key structures manage connections:

clientspendingwrite : per‑thread list of synchronous write requests.

clientspendingasyncwrite : per‑thread list of asynchronous write requests.

clientstoclose : global list of connections that need asynchronous closing.

Synchronous writes stay within the owning thread, while asynchronous writes use a pipe to hand off data to the appropriate worker. The pipe descriptors are declared as:

int fdCmdWrite; // write pipe
int fdCmdRead;  // read pipe

When a thread needs to send data to a client owned by another thread, it posts a message through the pipe (using AE_ASYNC_OP::CreateFileEvent) and the target thread adds the request to its write event loop.

Lock Mechanism

KeyDB implements a spin‑lock‑like primitive called fastlock . Its core data structures are:

struct ticket {
    uint16_t m_active; // unlock +1
    uint16_t m_avail;  // lock +1
};

struct fastlock {
    volatile struct ticket m_ticket;
    volatile int m_pidOwner; // owning thread id
    volatile int m_depth;   // recursion depth
};

Fastlock uses atomic operations ( __atomic_load_2, __atomic_fetch_add, __atomic_compare_exchange) to compare m_active and m_avail and decide whether the lock can be acquired. It provides two acquisition methods:

trylock : attempts once and returns immediately on failure.

lock : busy‑waits, yielding the CPU after a large number of spins (using sched_yield).

KeyDB combines trylock with its event loop so that a thread that cannot obtain the lock simply defers processing to the next epoll wait cycle.

Active‑Replica

KeyDB adds an active‑replica mode where replicas can accept writes. Each replica is identified by a UUID to avoid circular replication. A new rreplay API bundles incremental commands together with the replica’s UUID. Keys and values carry a timestamp‑based version (timestamp shifted left 20 bits plus a 44‑bit counter) that is used for conflict detection: if a local key has a newer version than an incoming write, the write is rejected.

Overall, the article provides a detailed walkthrough of KeyDB’s multithreaded design, connection handling, custom lock implementation, and active‑replica features.