Mastering Redis Persistence: Deep Dive into RDB, AOF, and Hybrid Strategies

Redis Persistence

Redis stores data in memory, which is fast but volatile; to avoid data loss after a restart, Redis persists data to disk using three methods: AOF (Append Only File), RDB (Redis DataBase snapshot), and a hybrid approach that combines both.

Persistence Process

Persistence Workflow

Client sends a write operation to the server.

Server receives the write request.

Server calls the write system call to write data to the OS buffer.

OS transfers the buffer to the disk controller.

Disk controller writes data to the physical medium.

Data Synchronization Mechanism

Redis uses a master‑slave replication model similar to relational databases. Reads are served by slaves, while writes are processed by the master and then propagated to slaves.

If every modification request were sent to different instances, maintaining consistency would require costly locking and coordination; handling all writes on the master avoids this overhead.

Redis Master‑Slave Sync Process Details

Assume two instances: master (172.16.19.3) and slave (172.16.19.5). To make the slave follow the master, run:

replicaof 172.16.19.3 6369

The sync consists of three phases:

Phase 1: Slave sends psync to master. Master replies with FULLERSYNC, providing its runID and current offset.

Phase 2: Master generates an RDB snapshot (via bgsave) and sends it to the slave, which loads it locally.

Phase 3: Master records new write commands in the replication buffer and streams them to the slave.

After these phases, the master and slave are synchronized.

Why Use RDB Instead of AOF for Initial Sync

RDB snapshots are compact binary files that can be quickly transferred and loaded, whereas AOF logs are larger text files that require more bandwidth and memory.

Master‑Slave Cascading Mode Reduces Master Load

When many slaves replicate directly from the master, the master must generate and transmit multiple RDB files, causing CPU and network pressure. Cascading replication lets a slave act as an intermediate source, offloading the master.

The master forks a child process to generate the RDB file; this fork blocks the main thread, but the child handles the snapshot without affecting client requests.

Full Sync Followed by Incremental Sync

After the initial FULLERSYNC, the master and slave maintain a long‑lived connection for incremental command propagation. If the network breaks, Redis can resume with incremental replication (Redis 2.8) or fall back to a full copy (Redis 2.8 and earlier).

Shared Buffer Design (Redis 7.0+)

Both master and slaves write commands to a shared replication buffer, eliminating duplicate data storage and reducing memory usage.

repl_backlog_buffer Extension

The circular repl_backlog_buffer stores recent write commands. If the buffer fills, older data is overwritten, which can cause inconsistencies for slow slaves. The repl_backlog_size parameter (typically twice the expected write rate) mitigates this risk.

RDB Mechanism

RDB creates a binary snapshot of the dataset at a point in time. The default file name is dump.rdb. Snapshots can be triggered manually with save (blocking) or asynchronously with bgsave (non‑blocking, uses fork and copy‑on‑write).

save Trigger

The save command blocks the server while writing the snapshot, which is unsuitable for large datasets.

bgsave

bgsave Trigger

bgsave

forks a child process that writes the RDB file, allowing the parent to continue serving clients. The child uses copy‑on‑write to handle concurrent writes.

Write‑Once Copy (Copy‑On‑Write)

During bgsave, the parent and child share memory pages. When the parent modifies data, a copy of the page is made for the child, ensuring the snapshot reflects the state at fork time.

Automatic Trigger

Configuration save lines in redis.conf define when Redis automatically performs bgsave, e.g., save 900 1 (once per 15 minutes if at least one key changed), save 300 10, and save 60 10000. Disabling all save lines turns off automatic snapshots. stop-writes-on-bgsave-error yes: stops writes if a snapshot fails. rdbcompression yes: compresses the snapshot. rdbchecksum yes: adds CRC64 checksum (≈10 % overhead). dbfilename dump.rdb and dir /path/to/dir set file name and directory.

RDB Advantages

Compact binary file, fast backup and disaster recovery.

Parent forks a child for I/O, keeping the main thread responsive.

Faster restore of large datasets compared to AOF.

RDB Disadvantages

Snapshot is a point‑in‑time copy; data modified after the fork is not persisted until the next snapshot, risking loss.

Frequent snapshots cause disk I/O and fork overhead, especially with large memory.

AOF Mechanism

AOF logs every write command to a file, enabling reconstruction by replaying the commands. The log format uses the Redis protocol (e.g., *3 \n$3\r\nSET\r\n$7\r\nmykey\r\n$5\r\nvalue\r\n).

Write‑After Log Advantages and Risks

Only successful commands are logged, avoiding syntax errors.

Logging after command execution prevents blocking the write operation.

Risk: a crash between command execution and log write can lose that command.

Risk: AOF writes are performed by the main thread; heavy disk sync can block subsequent operations.

AOF Trigger Mechanisms

always

: sync to disk after every write (high durability, low performance). everysec: buffer writes and sync every second (balanced). no: rely on OS to flush buffers (best performance, higher data loss risk).

File Rewrite

Because the AOF file grows over time, Redis rewrites it by creating a new compact file that contains a single command per key ("many‑to‑one" compression). The rewrite runs in a background child process ( bgrewriteaof) to avoid blocking the main thread.

The rewrite process copies the current dataset (via fork) and writes a new AOF while the parent continues logging new commands to a temporary buffer. After completion, the new file replaces the old one.

AOF Incremental Sync

Setting aof-rewrite-incremental-fsync yes makes Redis perform incremental fsync operations during AOF rewrite, reducing pause time for large files.

aof-rewrite-incremental-fsync yes

Pros and Cons

Pros

Very low data loss (typically ≤1 second with everysec).

High write performance when using everysec or no.

Human‑readable log enables manual recovery after accidental FLUSHALL.

Background rewrite does not block client operations.

Cons

AOF files are larger than RDB snapshots.

Recovery can be slow because every command must be replayed.

Historical bugs caused inconsistencies during AOF replay.

Hybrid Persistence (RDB + AOF)

Redis 4.0 introduced a hybrid mode that writes an RDB snapshot periodically and logs subsequent writes with AOF. This combines fast restart (RDB) with minimal data loss (AOF) while keeping the AOF file size manageable.

By adjusting auto-aof-rewrite-percentage and auto-aof-rewrite-min-size, Redis automatically rewrites the AOF when it grows beyond configured thresholds.

Overall, choosing between RDB, AOF, or the hybrid approach depends on the required balance between performance, restart speed, and data durability.

Source: https://www.cnblogs.com/jingzh/p/17292280.html (© original author)