Analysis of Redis Design: Network Model, Data Structures, Memory Management, Persistence, and Clustering

Song Zengkui, a Tencent engineer who joined in 2016, works on massive‑service backend design and development. He now leads the QQ group backend project, enjoys researching technology, and focuses on high‑concurrency architecture design and performance optimization.

In the second half of the year, he spent spare time studying parts of the Redis source code. This article analyzes Redis’s design from four perspectives: network model, data structures & memory management, persistence, and multi‑node collaboration. Readers are invited to point out any inaccuracies.

Redis is a widely used caching component. The most direct way to study a component framework is from the application side, examining each step of a request to quickly grasp the design philosophy. Using Redis as an example, the article describes how Redis handles network requests, organizes data structures, manages memory, dumps data to disk for persistence, and synchronizes across multiple machines.

1. Network Model

Redis follows a typical Reactor‑based event‑driven model: single‑process, single‑thread, similar to SPP’s asynchronous model.

The workflow is divided into three synchronous modules: request acceptor, response processor, and reply handler. Every request passes through these three stages.

Redis integrates multiple event‑management mechanisms such as libevent/epoll/kqueue/select, allowing the OS to choose the most suitable one; libevent is usually the optimal choice.

The event‑driven model is efficient and low‑cost, but long‑running operations block the event loop. For example, deleting all keys in Redis can stall all incoming requests. Understanding this model helps developers avoid long‑latency operations and keep most requests short to exploit the model’s strengths.

2. Data Structures and Memory Management

2.1 Strings

Structure

Redis strings are a second‑level wrapper around C strings. The structure is:

struct sdshdr {
    long len;
    long free;
    char buf[];
};

Besides the character buffer, Redis allocates extra fields to manage length and free space.

Memory Management

Redis uses dynamic memory allocation with strategies such as “pre‑allocation” and “lazy free” to reduce fragmentation and memory churn. When a string grows, the buffer is expanded (typically doubled or increased by a megabyte). When it shrinks, memory is reclaimed lazily.

These techniques balance space waste against time overhead, similar to STL, tcmalloc, or protobuf arena mechanisms.

Binary Safety

The length field (len) determines the end of the string, not a terminating ‘\0’, making Redis strings binary‑safe. Binary data such as protobuf‑encoded strings can be stored safely.

2.2 Dictionaries (Hashes)

Redis dictionaries are hash tables. They rely on MurmurHash2 for hashing, chain addressing for collision resolution, and a lazy rehash process to avoid long blocking operations.

Hash Algorithm

MurmurHash2 provides good distribution and fast computation. The author previously leveraged it in a shared‑memory local cache.

Collision Resolution

Redis uses chaining. With a high‑quality hash function like MurmurHash, chaining works efficiently.

Rehash (Resize)

Redis maintains the hash table’s load factor by lazily moving entries from the old table to a new one. A rehash index (rehashidx) marks the start; each subsequent operation migrates a small batch of entries until the old table is empty, avoiding a single long‑duration copy.

2.3 Integer Sets

Redis stores integers in a variable‑length encoding (16/32/64 bits). Insertion may trigger a scale upgrade, leading to O(n) complexity for the upgrade, which is a trade‑off between memory usage and time.

2.4 Skiplist

Redis uses a standard skiplist (called “skiplist”) mainly for sorted sets. It provides multi‑level linked lists for fast range queries.

2.5 Linked List

Redis implements a doubly‑linked, non‑circular list with head and tail pointers. Operations at the ends are O(1); lookups are O(n).

3. AOF and RDB Persistence

3.1 AOF Persistence

AOF logs write commands to an append‑only file. A timed event flushes the AOF buffer to disk.

3.2 AOF Rewrite

To shrink the AOF file, Redis creates a new AOF that contains only the minimal set of commands needed to reconstruct the current state. The rewrite runs in a child process to avoid blocking the main server.

3.3 RDB Persistence

SAVE and BGSAVE generate RDB snapshots. BGSAVE forks a child process, so the main server continues handling requests. Persistence occurs periodically or after a configured number of changes.

4. Multi‑Node and Cluster

4.1 Master‑Slave Replication

To avoid a single point of failure, Redis supports master‑slave replication. The original SYNC command performed full data copy each time; later PSYNC introduced partial resynchronization using offsets and a backlog buffer.

Partial Resynchronization (PSYNC)

Both master and slave maintain a sequence number. The slave sends its seq to the master; if the master’s backlog still contains the missing range, it serves the partial sync; otherwise a full sync is performed. This trades extra memory for faster incremental sync.

4.2 Sentinel

Sentinel adds monitoring and automatic failover for multiple master‑slave setups, ensuring high availability.

4.3 Cluster

Redis Cluster partitions data into slots; each node owns a subset of slots. Clients that request a key belonging to another node receive a MOVED error with the correct slot location, meaning the client must be cluster‑aware.

The overall takeaway is that every design decision in Redis is a trade‑off between speed, memory, and complexity. Understanding these compromises helps developers make informed optimizations in their own projects.