Skip to content
Redis Replication & Sentinel: Async Sync, Backlog Mechanics & Failover Internals
Databases 44 min read

Redis Replication & Sentinel: Async Sync, Backlog Mechanics & Failover Internals

This article dissects Redis master-slave replication and Sentinel failover through three real incidents, explaining the replid/offset/backlog trio for incremental sync, async replication trade-offs, Sentinel's quorum/majority election, brain-split risks, and production configuration formulas for backlog sizing and replication storm mitigation.

Architecture Pony
Architecture Pony
Architecture Pony

Introduction

Persistence solves data survival on a single machine restart, but what if the machine itself disappears — disk failure, data-center power loss, or accidental deletion? The ceiling of single-node reliability is hardware lifespan. The only way to break it is to keep data on multiple machines simultaneously and let them elect a new leader automatically. That is exactly what master-slave replication plus Sentinel achieve: replication handles "multi-active" data, Sentinel handles "who decides".

These two operational concepts hide three clever Redis designs:

The replid + offset + backlog trio bridges full and incremental sync, determining whether a disconnected replica catches up with a few hundred commands or re-transfers tens of gigabytes.

Replication is purely asynchronous — the master never waits for replica acknowledgments. This buys write performance but sows the seeds of data loss and split-brain.

Sentinel is not a cluster ; it is a group of communicating observers. The actual failover action is a single SLAVEOF NO ONE command sent by the elected leader to a chosen replica.

Three Incidents, Three Lessons

Incident 1: Five Replicas Restart Simultaneously, Master Crushed by Its Own

A deployment changed the replica image, causing five replicas to restart within 30 seconds. Each restart triggered PSYNC; because their offsets were stale, all fell back to full sync. The master forked five times, generated five 40 GB RDB files, and streamed them simultaneously. Outbound bandwidth saturated, fork pauses stacked, and write P99 latency jumped from 3 ms to 4.2 seconds. The application mistook this for master failure and triggered degradation. Lesson: a replica's failure can infect the master via full sync — the most counter-intuitive property of master-slave architecture.

Incident 2: Undersized Backlog Turns a Blip into Full Sync

The instance used the default repl-backlog-size of 1 MB. Write peak was 30k commands/sec, average command 60 bytes → ~1.8 MB/sec replication stream. The 1 MB ring buffer holds only 0.3 seconds of data. A 6-second switch blip caused the replica to reconnect with an offset already evicted from the backlog, forcing +FULLRESYNC. The blip itself caused no outage; the ensuing full sync did — and the cycle repeats.

Incident 3: Failover Completes, Old Master Returns as a Second Master

Sentinel down-after-milliseconds was 5000 ms. An 8-second network partition made all three Sentinels mark the master ODOWN and complete failover. Two minutes later the old master's network healed. Its config still said it was master, so it kept accepting writes for those two minutes. When Sentinels finally demoted it via full sync, those writes were overwritten by the new master's RDB. The split-brain window = time from failover completion to old master demotion; writes in that window are pure loss.

Technical Analysis: How Replication Connects

2.1 Three Topologies and Their Risks

Star (one master, many replicas): Lowest latency, freshest data. Cost: master outbound bandwidth and fork count grow linearly with replica count.

Chain (master → replica → replica): Distributes load to intermediate nodes. Cost: latency accumulates per hop; failure domain widens — if the middle node dies, all downstream replicas lose connection.

Tree (multi-level chain): Suits cross-datacenter: master replicates to same-city replica, which then replicates to remote replicas.

Selection criterion: on cross-datacenter links, do you fear saturating master bandwidth more, or data latency more?

Overlooked detail: replicas are read-only by default ( replica-read-only yes), but this is only a guard against accidental writes. Real protection comes from min-replicas-to-write on the master and funneling all writes through a single client entry point. Teams that disable read-only for "convenient data import" often harvest a split-brain master-replica pair during a later incident.

2.2 Ten Steps of a Replication Handshake

Many resources simplify replication establishment to "send PSYNC, receive RDB". The real process starts from TCP connection:

1  Replica connects to master:6379
2  Master replies PING → +PONG
3  AUTH if password/ACL configured
4  REPLCONF listening-port 6380
5  REPLCONF capa eof capa psync2
6  PSYNC <replid> <offset>
7  Master replies +FULLRESYNC replid offset
8  Master forks, streams RDB
9  Replica flushes old data, loads RDB, replies REPLCONF ACK
10 Enter command propagation, ACK every second

Step 5's capability negotiation is the watershed. capa eof means the replica supports length-less RDB stream (master sends RDB then command stream, separated by EOF marker). capa psync2 means it supports psync2 protocol, allowing reuse of replid for partial resync after replica restart. Older replicas lacking these capabilities force the master into fallback logic — many "configured psync2 but always full sync" issues stem from replica version being too old for capability negotiation to succeed.

2.3 Three Pillars: replid, offset, backlog

replid — "ID card of this dataset". Each master generates a 40-char hex random string at startup; all replicas replicated from it remember it.

offset — "byte cursor of replication stream". Master increments per byte written to replication stream; replica tracks how far it has processed.

backlog — fixed-size ring buffer. Master keeps a copy of recent bytes written to replication stream, for disconnected replicas to replay.

Relationship in one sentence: If replid matches, we share the same dataset; if offset falls within backlog's [start, end] range, the missing bytes can be supplied. Both conditions must hold for incremental sync; otherwise full sync. This also explains psync2's second replid2 — it records the previous master's replid before failover, paired with second_replid_offset, letting replicas continue on the new master without triggering full resync for the whole group.

2.4 Three Gates of Partial Resynchronization

Master runs masterTryPartialResynchronization upon receiving PSYNC; any gate failure → FULLRESYNC:

/* Gate 1: recognize replid */
if (strcasecmp(rid, server.replid) &&
    (server.replid2 == NULL ||
     strcasecmp(rid, server.replid2) ||
     psync_off > server.second_replid_offset))
    goto need_full_resync;

/* Gate 2: backlog covers offset */
if (!server.repl_backlog ||
    psync_off < server.repl_backlog_off ||
    psync_off > server.repl_backlog_off + server.repl_backlog_histlen)
    goto need_full_resync;

/* Gate 3: offset non-negative */
if (psync_off < 0) goto need_full_resync;

/* Success */
c->flags |= CLIENT_SLAVE;
addReplySds(c, sdscatprintf(... "+CONTINUE %s", server.replid));
replicationSetupSlaveForFullResync(c, ...);
return C_OK;

Gate 1 is often misread: it doesn't just compare replid strings. If the replica presents the previous master's replid (failover scenario), as long as its offset does not exceed second_replid_offset, continuation is allowed — that history is shared, data is consistent. This single design erased the cost of "full sync for all replicas after failover" and is a core reason to upgrade to Redis 4.0+.

2.5 Heartbeat: REPLCONF ACK Powers Three Functions

Command propagation is one-way (master → replica), but the replication link needs bidirectional liveness signals so the master knows if a replica is alive and how far it has replicated.

Replica sends REPLCONF ACK &lt;offset&gt; every second. This single packet does three things:

Tells master the replica is alive.

Reports replication progress (used by WAIT and min-replicas checks).

Allows Sentinels to calculate lag.

Based on this ACK, master can enforce "write availability check":

min-replicas-to-write 1
min-replicas-max-lag 10

Meaning: if the number of healthy replicas (lag ≤ 10 sec) drops below 1, master refuses writes and returns an error. This is Redis' official "reduce split-brain damage" mechanism — note the wording "reduce", not "prevent"; reasons detailed in section 5.3.

Core Processes: Full Sync and Incremental Sync Walkthrough

3.1 Full Sync: Eleven Moments

Full sync duration is often underestimated because it's not "transfer a file" but six serial stages: master fork + serialization + network transfer + replica flush + load + catch-up incremental . The slowest stage determines total time:

T+0  Master receives PSYNC, decides full sync required
T+0  replicationSetupSlaveForFullResync
     records repl_start_cmd_stream_on_ack
T+1  Fork child process rdbSaveToSlavesSockets
T+1  Main thread copies page tables, pauses (10 GB ≈ 260 ms)
T+2  Child serializes and writes to socket
T+2  Main process buffers subsequent writes into
     replica-specific output buffer
T+N  Replica finishes receiving RDB, flushes old data
T+N  Loads RDB (≈ 60%-80% of snapshot time)
T+N  Replica sends REPLCONF ACK to start command stream
T+N  Replays buffered incremental commands, catches up
T+N  Enters steady state, ACK every second

Critical: watch the replica-specific output buffer from T+2 to T+N. New writes during full sync are not lost; they are buffered per replica (controlled by client-output-buffer-limit replica, default 256 MB/64 MB/60 s). A 40 GB RDB taking 20 minutes with 200 MB of new writes means this buffer must hold 200 MB. If write rate bursts and overflows the buffer, master proactively disconnects that replica , which then reconnects and triggers another full sync — a death spiral common in "large instance + high write + slow network" combos.

3.2 Incremental Propagation: One Command, Three Forwards

In steady state, each write command is serialized to RESP and pushed to three consumers:

All replica connections

Backlog ring buffer

AOF (if enabled)

Core entry: propagateNow → replicationFeedSlaves. Notably, it writes to backlog first, then to each replica's buffer — this order guarantees "if a command is in backlog, it has a definite offset for alignment".

Backlog stores raw RESP command bytes, not parsed structures. So replica replay just feeds bytes to the command parser; cost is backlog capacity must be calculated in bytes, not command count. That's why repl-backlog-size unit is bytes and must be estimated with average command length.

Core Source Code: Backlog, Sentinel, State Machine

4.1 Ring Buffer: Overwrite Oldest When Full

repl_backlog

is a fixed-size contiguous memory region. repl_backlog_idx points to next write position; repl_backlog_off points to global offset of oldest byte in buffer. Writes overwrite circularly:

while (len) {
    size_t thislen = len;
    size_t avail = server.repl_backlog_size - server.repl_backlog_idx;
    if (thislen > avail) thislen = avail;
    memcpy(server.repl_backlog + server.repl_backlog_idx, p, thislen);
    server.repl_backlog_idx += thislen;
    if (server.repl_backlog_idx == server.repl_backlog_size)
        server.repl_backlog_idx = 0;
    p += thislen; len -= thislen;
}
repl_backlog_histlen

tracks current valid bytes (max = size). repl_backlog_off = master total offset − histlen. When a disconnected replica requests continuation, master computes how far its offset is from the oldest byte, then reads from that position in ring order:

skip = offset - server.repl_backlog_off;
j = (server.repl_backlog_idx + server.repl_backlog_size
     - (server.repl_backlog_histlen - skip)) % server.repl_backlog_size;
while (len > 0) {
    thislen = j+len > size ? size-j : len;
    addReplyProto(c, backlog + j, thislen);
    j = (j+thislen) % size;
    len -= thislen;
}

The modulo arithmetic handles wrap-around by splitting into two copies. Understanding this makes backlog metrics meaningful — production hard rule: if histlen / size chronically exceeds 80%, increase backlog.

4.2 Sentinel Step One: Subjective vs Objective Down

Every down-after-milliseconds, each Sentinel pings master, replicas, and other Sentinels. Timeout → SDOWN (Subjective Down) — a single Sentinel's private opinion, possibly due to its own network issue; cannot trigger failover .

To escalate to ODOWN (Objective Down) , the Sentinel must poll others:

void sentinelCheckSubjectivelyDown(ri) {
    if (now - ri->last_pong_time > ri->down_after_period) {
        if (ri->flags & SRI_MASTER && info_refresh > down_after)
            ri->flags |= SRI_S_DOWN;
    }
}

void sentinelCheckObjectivelyDown(ri) {
    quorum = ri->quorum;
    count = 1;  /* self vote */
    for (each other sentinel s) {
        if (s also reports master down in this epoch)
            count++;
    }
    if (count >= quorum)
        ri->flags |= SRI_O_DOWN;
}

quorum is per-master in sentinel.conf, not a cluster parameter. Three Sentinels, quorum=2 means "at least two Sentinels think it's down". This is separate from "how many votes to become leader" — the latter uses majority ( > total/2 ). Mixing these two numbers is the most common Sentinel misconfiguration : setting quorum=1 in a three-node cluster lets a single Sentinel's mistake trigger failover because ODOWN only needs 1 vote.

4.3 Who Fails Over: Raft-Style Vote with config epoch

After ODOWN, the discoverer doesn't act alone. It votes for itself first, then sends SENTINEL is-master-down-by-addr to others to solicit votes . Each Sentinel votes once per epoch, first-come-first-served.

Elected leader requires votes ≥ majority (total/2 + 1) AND ≥ quorum. This mirrors Raft election, except it runs only on failure, no periodic heartbeats for term maintenance.

epoch prevents "two Sentinels elected in different rounds". Each failover attempt increments config epoch; votes carry epoch; old-epoch votes are void in new epoch. Without epoch, two Sentinels in two independent misjudgments could each gather two votes and produce two masters. This is why Sentinels must intercommunicate — not for monitoring, but to reach unique consensus.

4.4 Choosing the New Master: Three-Level Sort, One Tie-Breaker

sentinelSelectSlave

first excludes replicas that are:

SDOWN

Disconnected too long replica-priority = 0 (never candidate, used for datacenter weight)

Unresponsive for 5 seconds

INFO not refreshed

Then three-level sort:

Lower priority wins.

Higher slave_repl_offset wins (most up-to-date data).

Lexicographically smaller runid wins.

Level 3 seems arbitrary but ensures determinism : if two replicas tie on priority and offset, different Sentinels must pick the same one, otherwise their decisions diverge. Using runid as final tie-breaker guarantees "same input → same output". Practical tip: to make a specific replica preferred, don't just tweak priority (may conflict with offset); ensure it has the highest offset — i.e., lowest replication lag.

4.5 Failover State Machine: Seven States

After election, leader enters a fixed state machine with timeouts and retries. Understanding this chain decodes log events like +switch-master, +promoted-slave, +slave-reconf-sent:

WAIT_START        Confirm ODOWN, epoch +1
SELECT_SLAVE      Pick new master, retry on failure
SEND_SLAVEOF_NOONE Send REPLICAOF NO ONE to new master
WAIT_PROMOTION    Poll until role:master
RECONF_SLAVES     Remaining replicas repoint to new master
UPDATE_CONFIG     Rewrite local config, broadcast +switch-master
(Old master recovers) Detect role still master, send REPLICAOF to demote

The last step is not in the state machine but part of Sentinel's steady monitoring. When old master recovers, Sentinel sees role:master in its INFO while local config says it should be replica, so it sends SLAVEOF to demote. This demotion follows the standard full-sync path — old master's writes during the split-brain window are overwritten by new master's RDB, no merge, no warning. This is the key to understanding "split-brain loss is irreversible".

Advanced Topics: Four Accounts You Must Calculate Upfront

5.1 How Big Should Backlog Be?

Single formula:

backlog ≥ write_rate (bytes/sec) × max_tolerable_disconnect_sec × safety_factor_2

Write rate is not QPS; derive from

INFO stats
instantaneous_input_kbps

or watch master_repl_offset growth. Example: offset grows 108 MB/min → 1.8 MB/s. Tolerate 60 sec disconnect → backlog ≥ 1.8 × 60 × 2 ≈ 216 MB. Default 1 MB is only for demos; every production instance must set explicitly.

Cost: backlog consumes master memory. Changing repl-backlog-size doesn't take effect until next full sync or restart. So calculate before launch, not after incident. Also, backlog is allocated only when at least one replica exists ( repl-backlog-ttl controls retention after last replica disconnects, default 3600 sec).

5.2 Three Forms of Replication Storm and Fixes

Multiple replicas full-sync simultaneously. Fix: repl-diskless-sync-delay 5 — master waits 5 seconds after first full-sync request, merges other replicas arriving in that window into the same fork and RDB stream. N full syncs become 1. Default is 5; many change to 0 for "speed", actively abandoning merge.

Replica-of-replica storm. In chain replication, intermediate node restart triggers its own full sync and cascades downstream.

Master restart triggers cluster-wide full sync. Because replid changes, all replicas cannot continue (unless psync2 enabled and replid2 matches).

Often overlooked switch: repl-diskless-sync yes (diskless replication) lets forked child write RDB directly to socket, saving two disk I/O trips. Huge gain when disk is slow, network fast; worse when network slow — child blocks on socket write, forked memory pages not released. Decision criterion: Can disk sequential write speed keep up with network bandwidth? If yes, use disk replication; if no, use diskless.

5.3 Split-Brain: Why You Can Only Reduce Damage

Split-brain essence: "Old master doesn't know it's old." During failover, old master may be isolated from Sentinels but still connected to clients — it keeps accepting writes that never reach new master. min-replicas-to-write logic: if I can't see enough healthy replicas, I might be partitioned, so I refuse writes. When isolated, old master indeed sees no replicas, so it rejects writes, compressing split-brain window to "from partition to parameter effect" — tens of seconds.

But it cannot eliminate split-brain, for two reasons:

Check happens at write command arrival; window exists between detection and refusal.

If partition isolates old master from Sentinels but not from replicas, old master still sees healthy replicas and continues accepting writes.

Network topology diversity means no single parameter covers all partition shapes. True fix requires client-side consistency reads (verify epoch after failover) or stronger consensus storage. Realistic engineering goal: not "zero loss" but "loss measurable and compensable" — treat split-brain window duration as part of SLA, add reconciliation jobs, rather than chasing a non-existent perfect parameter.

5.4 Six Hard Rules for Sentinel Deployment

At least three, odd number. Majority = total/2+1. With two nodes, majority=2 (unanimous); one down → failover impossible (availability worse than single Sentinel). Three nodes: majority=2, can lose one.

Don't co-locate with data nodes. Same host failure kills master and Sentinel together; remaining Sentinels may not reach quorum.

Spread three Sentinels across three failure domains. Same rack, same switch = one domain.

Distinguish quorum vs majority. Quorum decides ODOWN; majority decides leader election.

Clients must speak Sentinel protocol (ask Sentinels for master address, then connect), and handle +switch-master reconnection. Plain connection pools pointing at static IP will stick to old master forever after failover.

Sentinels must persist their config. sentinel.conf is auto-rewritten (records current master). File must be writable; otherwise restart reverts config, possibly triggering second failover.

5.5 Replication Lag Troubleshooting Order

Lag has no single metric; combine three views:

Byte view: master_repl_offset - slave_repl_offset difference.

Time view: Replica's master_last_io_seconds_ago from INFO replication.

Capacity view: Replica's output buffer vs client-output-buffer-limit replica.

Fixed investigation order:

Is byte difference growing continuously? (Bandwidth or replica processing bottleneck)

Is replica CPU stuck on slow queries / big keys?

Is master in full sync or fork pause?

Stable difference of a few hundred KB is normal; monotonic increase is the problem.

5.6 Redis 7 Key Change

Redis 7 changed replication buffer from "one giant shared buffer for all replicas" to shared backlog + per-replica incremental buffers . Benefit: a slow replica's backlog pressure only affects its own buffer, no longer drags down other replicas or causes them to disconnect. Solves the long-standing "one slow replica drags down the whole group" issue. Trade-off: memory accounting more complex — must calculate per replica. This makes tuning client-output-buffer-limit replica more critical: it's now a per-replica threshold.

Learning Path

Layer 1 (Configure & Use): Spin up 1 master + 2 replicas; verify INFO replication fields; kill replica process, restart, observe full vs incremental; set backlog to 100 KB, retry, watch fallback.

Layer 2 (Calculate & Tune): Compute backlog via formula; load-test "simultaneous replica restart", measure master fork pause and bandwidth curves; benchmark repl-diskless-sync on/off.

Layer 3 (Diagnose & Recover): Use iptables to cut master-Sentinel link (keep master-replica), observe split-brain, test min-replicas-to-write; then cut master-replica links, observe post-failover old master data overwrite.

Layer 4 (Modify Source): Read replication.c and sentinel.c; manually trace psync2's replid2 logic to understand why failover avoids cluster-wide full sync.

Interview Questions (Summary)

The article includes 20 Q&A pairs covering: async vs sync replication, why RDB not AOF for full sync, PSYNC vs SYNC, offset definition and location, backlog overflow behavior, replica restart full sync cause, replica writability and expired key handling, accurate lag measurement, min-replicas-to-write limitations, Sentinel node count and oddness, SDOWN vs ODOWN, ODOWN quorum mechanism, new master selection rules, config epoch purpose, client experience during failover, old master demotion process, multi-master monitoring, chain replication risks, repl-diskless-sync-delay rationale, and production default config checklist.

One-Page Cheat Sheet

Replication = async, master doesn't wait for replicas<br>Three pillars: replid + offset + backlog<br>Incremental three gates: verify id, cover offset, non-negative<br>Backlog = rate × tolerable_seconds × 2<br>Full sync pain: fork + per-replica buffer<br>SDOWN = private opinion, ODOWN ≥ quorum<br>Leader needs majority, not quorum<br>Master selection: priority → offset → runid<br>Old master demotion = full sync, dual-master writes always lost

Conclusion

The capability boundary of master-slave plus Sentinel compresses to one sentence: It provides "automatic availability recovery", not "zero data loss". Replication is asynchronous; failover takes seconds to tens of seconds; writes in the split-brain window get overwritten — these are design premises, not implementation bugs. Understanding them turns configuration from "copying templates" into "calculating parameters by SLA": backlog sized by tolerable disconnect time, down-after by acceptable false-positive rate, quorum and majority by failure domain layout.

Original Source

Signed-in readers can open the original source through BestHub's protected redirect.

Sign in to view source
Republication Notice

This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactadmin@besthub.devand we will review it promptly.

High AvailabilityRedisreplicationSentinelFailoverBacklogSplit BrainPSYNC
Reader feedback

How this landed with the community

Sign in to like

Rate this article

Was this worth your time?

Sign in to rate
Discussion

0 Comments

Thoughtful readers leave field notes, pushback, and hard-won operational detail here.