Redis Architecture Overview: Introduction, Memory Model, Persistence, Replication, and Cluster Modes

Redis (Remote Dictionary Server) is an open‑source, in‑memory key‑value store written in ANSI C and licensed under the BSD license. It supports strings, hashes, lists, sets, and sorted sets, and provides APIs for many programming languages.

The database was created in 2009 by Salvatore Sanfilippo (Antirez) to solve load problems for a personal website.

1. Memory Model

Redis’s memory usage can be inspected with the INFO MEMORY command. Important metrics include:

used_memory : total memory allocated by Redis (including virtual memory).

used_memory_rss : actual memory used by the Redis process as seen by the OS (includes process overhead, excludes virtual memory).

mem_fragmentation_ratio : ratio of used_memory_rss to used_memory, indicating fragmentation.

mem_allocator : the memory allocator in use, typically jemalloc.

Memory is divided into four main parts:

Data : the actual key‑value data, counted in used_memory.

Process overhead : memory required for the Redis process itself, not counted in used_memory.

Buffer memory : client buffers, replication backlog, and AOF buffer ( aof_buf).

Memory fragmentation : overhead caused by frequent allocation and deallocation.

Redis objects are represented by redisObject structures, which contain fields such as type, encoding, lru, refcount</strong>, and <code>ptr. The type field identifies the data type (string, list, hash, set, sorted set). refcount tracks how many references exist to an object, enabling shared objects for frequently used integer strings.

Redis uses the Simple Dynamic String (SDS) library instead of plain C strings. SDS stores the length and free space, providing O(1) length queries, automatic buffer resizing, and safe binary data handling.

2. Persistence (RDB & AOF)

Redis offers two persistence mechanisms:

RDB : snapshots of the dataset written to disk at configurable intervals using the BGSAVE command. The process forks a child that writes the RDB file while the parent continues serving clients.

AOF (Append‑Only File) : logs every write command. The AOF buffer ( aof_buf) is flushed to disk according to the appendfsync policy (always, everysec, or no). A background rewrite ( BGREWRITEAOF) compacts the AOF file.

Key AOF policies: always: every write is fsynced immediately – highest durability but lowest performance. everysec: default; writes are flushed every second – a balance of durability and speed. no: writes are only written to the OS cache – fastest but risk of data loss.

Below is a simplified excerpt of the Redis AOF flushing implementation (Redis 3.0):

void flushAppendOnlyFile(int force) {
    ssize_t nwritten;
    int sync_in_progress = 0;
    mstime_t latency;

    if (sdslen(server.aof_buf) == 0) return;

    if (server.aof_fsync == AOF_FSYNC_EVERYSEC)
        sync_in_progress = bioPendingJobsOfType(REDIS_BIO_AOF_FSYNC) != 0;

    if (server.aof_fsync == AOF_FSYNC_EVERYSEC && !force) {
        if (sync_in_progress) {
            if (server.aof_flush_postponed_start == 0) {
                server.aof_flush_postponed_start = server.unixtime;
                return;
            } else if (server.unixtime - server.aof_flush_postponed_start < 2) {
                return;
            }
            server.aof_delayed_fsync++;
            redisLog(REDIS_NOTICE,"Asynchronous AOF fsync is taking too long (disk is busy?). Writing the AOF buffer without waiting for fsync to complete, this may slow down Redis.");
        }
    }
    latencyStartMonitor(latency);
    nwritten = write(server.aof_fd, server.aof_buf, sdslen(server.aof_buf));
    latencyEndMonitor(latency);
    if (sync_in_progress) {
        latencyAddSampleIfNeeded("aof-write-pending-fsync", latency);
    } else if (server.aof_child_pid != -1 || server.rdb_child_pid != -1) {
        latencyAddSampleIfNeeded("aof-write-active-child", latency);
    } else {
        latencyAddSampleIfNeeded("aof-write-alone", latency);
    }
    latencyAddSampleIfNeeded("aof-write", latency);
    server.aof_flush_postponed_start = 0;
    if (nwritten != (signed)sdslen(server.aof_buf)) {
        static time_t last_write_error_log = 0;
        int can_log = 0;
        if ((server.unixtime - last_write_error_log) > AOF_WRITE_LOG_ERROR_RATE) {
            can_log = 1;
            last_write_error_log = server.unixtime;
        }
        if (nwritten == -1) {
            if (can_log) {
                redisLog(REDIS_WARNING,"Error writing to the AOF file: %s", strerror(errno));
                server.aof_last_write_errno = errno;
            }
        } else {
            if (can_log) {
                redisLog(REDIS_WARNING,"Short write while writing to the AOF file: (nwritten=%lld, expected=%lld)", (long long)nwritten, (long long)sdslen(server.aof_buf));
            }
            if (ftruncate(server.aof_fd, server.aof_current_size) == -1) {
                if (can_log) {
                    redisLog(REDIS_WARNING,"Could not remove short write from the append-only file. Redis may refuse to load the AOF the next time it starts. ftruncate: %s", strerror(errno));
                }
            } else {
                nwritten = -1;
            }
            server.aof_last_write_errno = ENOSPC;
        }
        if (server.aof_fsync == AOF_FSYNC_ALWAYS) {
            redisLog(REDIS_WARNING,"Can't recover from AOF write error when the AOF fsync policy is 'always'. Exiting...");
            exit(1);
        } else {
            server.aof_last_write_status = REDIS_ERR;
            if (nwritten > 0) {
                server.aof_current_size += nwritten;
                sdsrange(server.aof_buf, nwritten, -1);
            }
            return;
        }
    } else {
        if (server.aof_last_write_status == REDIS_ERR) {
            redisLog(REDIS_WARNING,"AOF write error looks solved, Redis can write again.");
            server.aof_last_write_status = REDIS_OK;
        }
    }
    server.aof_current_size += nwritten;
    if ((sdslen(server.aof_buf) + sdsavail(server.aof_buf)) < 4000) {
        sdsclear(server.aof_buf);
    } else {
        sdsfree(server.aof_buf);
        server.aof_buf = sdsempty();
    }
    if (server.aof_no_fsync_on_rewrite && (server.aof_child_pid != -1 || server.rdb_child_pid != -1))
        return;
    if (server.aof_fsync == AOF_FSYNC_ALWAYS) {
        latencyStartMonitor(latency);
        aof_fsync(server.aof_fd);
        latencyEndMonitor(latency);
        latencyAddSampleIfNeeded("aof-fsync-always", latency);
        server.aof_last_fsync = server.unixtime;
    } else if (server.aof_fsync == AOF_FSYNC_EVERYSEC && server.unixtime > server.aof_last_fsync) {
        if (!sync_in_progress) aof_background_fsync(server.aof_fd);
        server.aof_last_fsync = server.unixtime;
    }
}

Redis also supports background AOF rewriting to compress the log file. The rewrite creates a new AOF by iterating over the in‑memory dataset, similar to an RDB snapshot.

3. Replication

Replication consists of three stages: connection establishment, data synchronization, and command propagation.

Full sync : the master creates an RDB snapshot via BGSAVE, streams it to the replica, and then sends all buffered write commands.

Partial sync : the master and replica maintain a replication offset and a fixed‑size backlog buffer (default 1 MiB). If the replica reconnects within the backlog window, only missing commands are transferred.

During propagation, the master periodically sends PING to replicas, and replicas acknowledge receipt with REPLCONF ACK {offset} once per second.

4. Architecture Patterns

Sentinel mode provides high availability by monitoring masters and slaves, performing automatic failover, and notifying clients of the new master. Sentinels exchange PING and INFO commands, mark instances as subjectively or objectively down, and run elections when needed.

Cluster mode solves the single‑node limitation by sharding data across multiple nodes using virtual hash slots (0‑16383). Each key’s CRC16 hash determines its slot, and each node owns a subset of slots. The cluster supports online scaling by moving slots between nodes, and it handles request redirection when a key’s slot resides on a different node.

Overall, Redis combines an efficient in‑memory data model, flexible persistence options, robust replication, and scalable high‑availability architectures, making it suitable for caching, real‑time analytics, and many other use cases.