KeyDB’s Multithreaded Redis: Thread Model, Locks, and Active‑Replica Explained

Overview

KeyDB is a fork of Redis that retains 100% API compatibility while converting the originally single‑threaded KV store into a multithreaded system.

Thread Model

KeyDB splits Redis’s original main thread into a main thread and multiple worker I/O threads. Each worker thread listens on the same port (using SO_REUSEPORT), accepts connections, reads data and parses the protocol. Workers are bound to specific CPUs via SO_INCOMING_CPU. After parsing, a global lock protects access to shared memory.

The main thread is also a worker (index 0) and performs tasks that only the main thread can handle, such as serverCron.

Statistics processing

Client connection management

Database resize and reshard

AOF handling

Replication master‑slave synchronization

Cluster‑mode tasks

Connection Management

In Redis all connections are managed by a single thread; KeyDB assigns each worker its own set of connections, keeping creation, processing and destruction within the same thread. A new field int iel records the event‑loop index handling the connection.

KeyDB maintains three key structures for connection management: clients_pending_write: per‑thread list of connections awaiting synchronous writes. clients_pending_asyncwrite: per‑thread list for asynchronous writes (e.g., Pub/Sub messages sent from a different thread). clients_to_close: global list of connections that need asynchronous closing.

When a thread needs to send data to a client owned by another thread, it writes a message to a pipe using fdCmdWrite and fdCmdRead. The target thread receives the pipe message, registers a write event, and processes the send.

A global asynchronous close list handles client shutdowns that cannot be performed in the owning thread.

Lock Mechanism

KeyDB implements a spin‑lock‑like mechanism called fastlock. Its core structures are:

struct ticket {
    uint16_t m_active; // unlock +1
    uint16_t m_avail;  // lock +1
};
struct fastlock {
    volatile struct ticket m_ticket;
    volatile int m_pidOwner; // thread ID holding the lock
    volatile int m_depth;    // recursion depth
};

Atomic operations ( __atomic_load_2, __atomic_fetch_add, __atomic_compare_exchange) compare m_active and m_avail to decide lock acquisition. Two acquisition methods are provided: try_lock: returns immediately on failure. lock: busy‑waits, then yields with sched_yield after 1 048 576 attempts.

KeyDB combines try_lock with the event loop: a client’s lock is attempted before reading data; if it fails, the thread skips processing and will retry in the next epoll_wait cycle.

Active‑Replica

KeyDB supports multi‑active replicas, allowing each replica to be writable. Key features include:

Each replica has a UUID to avoid circular replication.

New rreplay API packages incremental commands with the replica’s UUID.

Keys and values carry a timestamp version; writes with older timestamps are rejected. The version is generated by left‑shifting the current timestamp 20 bits and adding a 44‑bit increment.

Project

https://github.com/JohnSully/KeyDB