Redis High‑Availability Deep Dive: Master‑Slave Replication, Sentinel, and Split‑Brain Protection

Why a Single‑Node Redis Is a Single‑Point‑of‑Failure

Even with perfect persistence, a lone Redis instance can become unavailable if the host crashes, the network drops, or the disk fails, causing the cache layer to collapse and all traffic to hit the database.

Running multiple instances as backups raises four technical questions:

How to keep data consistent across nodes?

Who detects a master failure and promotes a replica?

How do clients discover the new master?

Will data be lost during a switch?

The solution proceeds in three stages: master‑slave replication, Sentinel for automatic failover, and split‑brain protection.

1. Master‑Slave Replication

What Replication Does

The master handles writes; replicas copy the master’s data and serve reads, providing read‑write separation and redundancy.

Key insight: Replication is asynchronous by default, so the master returns to the client before replicas acknowledge. This yields high performance but introduces replication lag and a small risk of data loss during failover.

Quick Local Setup (3 ports: 6379 master, 6380 & 6381 replicas)

# Prepare three data directories
mkdir -p /tmp/redis-ha/6379 /tmp/redis-ha/6380 /tmp/redis-ha/6381
# Start the master
redis-server --port 6379 --dir /tmp/redis-ha/6379 --daemonize yes
# Start two replicas
redis-server --port 6380 --dir /tmp/redis-ha/6380 --daemonize yes
redis-server --port 6381 --dir /tmp/redis-ha/6381 --daemonize yes
# Make the replicas follow the master
127.0.0.1:6380> REPLICAOF 127.0.0.1 6379
127.0.0.1:6381> REPLICAOF 127.0.0.1 6379

Verification:

# Write on the master
127.0.0.1:6379> SET k hello
OK
# Read immediately on a replica
127.0.0.1:6380> GET k
"hello"
# Attempt to write on a replica (read‑only by default)
127.0.0.1:6380> SET k2 v2
(error) READONLY You can't write against a read only replica.
# View replication status on the master
127.0.0.1:6379> INFO replication
role:master  connected_slaves:2 ...

Key Replication Concepts

replid : 40‑byte random ID generated by the master at startup. Replicas store the master’s replid; on reconnection the master compares replids to decide full or incremental sync.

offset : Byte count of the replication stream written by the master ( master_repl_offset) and processed by the replica ( slave_repl_offset). The difference shows how far the replica lags.

repl_backlog_buffer : Fixed‑size circular buffer on the master that keeps recent replication bytes. If a replica reconnects while its missing range is still in the backlog, the master can send only the delta (incremental sync). If the backlog has overwritten the needed range, a full sync is required.

Full Sync (first connection)

Replica sends PSYNC ? -1 (unknown replid, no data).

Master replies with +FULLRESYNC <replid> <offset> and records those values.

Master forks a child to create an RDB snapshot ( bgsave) while still accepting writes.

RDB file is transferred to the replica.

Replica clears its data and loads the RDB, becoming an exact copy of the master at the fork moment.

Master streams the commands that accumulated in the replication buffer during the RDB creation; the replica executes them to reach full consistency.

Normal command propagation begins.

Incremental Sync (reconnection)

Replica remembers the last processed offset (e.g., 100).

Master continues to accept writes, advancing its offset (e.g., to 160).

Master keeps the new bytes in repl_backlog_buffer.

Replica reconnects with PSYNC <replid> 100 asking for data from offset 101.

Master checks that the replid matches and that offset 101‑160 is still in the backlog.

If both true → sends +CONTINUE and streams the missing bytes (incremental sync).

If either check fails → falls back to +FULLRESYNC (full sync).

Replica acknowledges the new offset with REPLCONF ACK 160.

When Full vs Incremental Sync Happens

First connection ( PSYNC ? -1) → Full sync.

Reconnection, same replid and offset still in backlog → Incremental sync.

Reconnection after long outage, backlog overwritten → Full sync.

Reconnection after master change (new replid) → Full sync.

2. Sentinel Mechanism: Automatic Master Switch

Replication solves data redundancy, but manual promotion of a replica after a master crash is slow and error‑prone. Sentinel automates detection, voting, promotion, and client notification.

Three Core Duties

Monitoring – periodic PING to all masters, replicas, and other sentinels.

Leader Election – when a master is deemed down, the remaining sentinels vote to elect a leader that will perform the failover.

Notification – the new master’s address is broadcast to replicas and clients.

Monitoring Tasks (different periods)

Every 1 s: PING to every node. If no reply within down-after-milliseconds, the node is marked subjectively down.

Every 10 s: INFO replication on the master to auto‑discover topology.

Every 2 s: Publish/subscribe on __sentinel__:hello so sentinels discover each other and exchange their view of the master.

Subjective vs Objective Down

Subjective Down (SDOWN) : a single sentinel marks the master down after missing PING responses.

Objective Down (ODOWN) : the sentinel that detected SDOWN asks other sentinels; if quorum of them agree, the master is considered objectively down and failover proceeds.

Leader Election and Failover Steps

Detection – sentinels ping the master every second.

Subjective down – a sentinel marks the master SDOWN and asks peers.

Objective down – when quorum peers agree, ODOWN is set.

Leader election – sentinels run a Raft‑like vote; the leader must obtain >50 % of total votes and at least quorum votes.

New master selection – the leader filters unhealthy replicas, then chooses the one with the lowest replica‑priority, highest offset, and finally smallest runid.

Promotion – the leader sends SLAVEOF NO ONE to the chosen replica, making it a master.

Re‑configuration – the leader tells the remaining replicas to follow the new master with REPLICAOF <new‑master‑ip> <port>.

Notification – +switch-master event is published; clients subscribed to Sentinel receive the new address.

Old master recovery – once the old master comes back, sentinels keep monitoring it; the old master is turned into a replica of the new master to avoid split‑brain.

Why three sentinels? With only two, a single failure leaves only one sentinel, which cannot obtain a majority (> ½) for leader election, causing the failover to stall. An odd number (≥ 3) guarantees a majority.

Sample Sentinel Configuration (one file per sentinel)

# sentinel-26379.conf
port 26379
daemonize yes
dir /tmp/redis-ha/sentinel-26379
sentinel monitor mymaster 127.0.0.1 6379 2   # quorum = 2
sentinel down-after-milliseconds mymaster 5000
sentinel failover-timeout mymaster 60000
sentinel parallel-syncs mymaster 1

Start three sentinels on different ports (or machines) with the same monitor line but different port and dir values.

3. Split‑Brain (Network Partition) and Data Loss

During a network partition the master may appear down to the sentinels, which promote a replica. The original master continues to accept writes, creating two masters. When the partition heals, the old master becomes a replica, discards its divergent data, and the writes performed during the split are lost.

Protection via Configuration

# redis.conf (on the master)
min-replicas-to-write 1   # require at least one healthy replica before accepting writes
min-replicas-max-lag 10    # a replica is considered healthy only if its last ACK is ≤10 s old

When the master loses all healthy replicas (e.g., during a split), it refuses writes, preventing loss of data that could not be replicated.

Optional Semi‑Synchronous Write with WAIT

# Write a key and wait up to 1000 ms for at least one replica to acknowledge
SET order:1 paid
WAIT 1 1000   # returns the number of replicas that confirmed

WAIT

reduces the window where a write could be lost, but it is not a full transaction guarantee.

4. Production Usage Boundaries

Single‑instance fits in memory, need automatic failover → Yes . Master‑replica + Sentinel is designed for this.

Read‑heavy, write‑light, want read scaling → Yes (accept read lag). Replicas can serve reads, but may be stale.

Data exceeds one machine’s memory → No . Sentinel does not shard; use Redis Cluster.

Write throughput exceeds a single master → No . Sentinel only switches masters; it does not distribute writes.

Strict consistency, zero data loss → No . Asynchronous replication can lose a few writes.

Local development or temporary cache → Not required . Single‑node Redis or persistence may be enough.

Production Recommendations

Deploy at least 1 master, 2 replicas, and 3 odd‑numbered sentinels.

Distribute sentinels across different machines or failure domains.

Clients must use Sentinel‑aware libraries and configure multiple sentinel addresses plus the masterName.

Enable persistence on the master to avoid empty data after a restart.

Configure min-replicas-to-write and min-replicas-max-lag according to tolerance.

Practice failover drills before production launch.

5. Common Issues and Troubleshooting

Replication Lag

Symptom: replica returns stale data.

Cause: asynchronous replication + network load or RDB loading.

Fix: monitor master_repl_offset vs slave_repl_offset; for strong consistency read from the master.

Frequent Full Syncs

Cause: repl-backlog-size too small or unstable network.

Fix: increase repl-backlog-size, improve network stability.

Sentinel Not Switching

Possible reasons:

Insufficient sentinel count (no majority). quorum set too high.

Network partition among sentinels.

No healthy replica to promote.

Client Still Points to Old Master

Cause: client hard‑codes master IP.

Solution: configure client with sentinel addresses and masterName; libraries (JedisSentinelPool, Lettuce, Redisson) handle +switch‑master events.

Write Failures During Failover

During the short window of detection, election, and promotion, writes may be rejected. Applications should implement retries and appropriate timeouts.

6. Client Connection Details

Clients never talk to Sentinel for data; Sentinel is only a discovery and notification service.

On startup, the client contacts one or more sentinels to ask SENTINEL get-master-addr-by-name mymaster for the current master address.

The client then opens a normal Redis connection to that master (or a replica for reads).

When Sentinel publishes +switch-master, the client library re‑queries the master address and reconnects.

Typical Java configuration:

# Application config
sentinel-1:26379
sentinel-2:26379
sentinel-3:26379
masterName = mymaster
# At runtime the library connects to the current master IP:port

Appendix: Replication Output Buffer vs repl_backlog_buffer

Each replica connection has its own TCP output buffer (limited by client-output-buffer-limit replica). If a replica is slow, the buffer grows and may be closed when limits are exceeded.

The global repl_backlog_buffer is a circular buffer on the master used only for replaying missed data to reconnecting replicas. Its size is controlled by repl-backlog-size and, when full, older data is overwritten.

# Example limits
client-output-buffer-limit replica 256mb 64mb 60   # hard 256 MB, soft 64 MB for 60 s
repl-backlog-size 64mb                         # size of the circular backlog

When the backlog is too small, a replica that stays offline for a while will be forced into a full sync, increasing master load.

Eventually, when a single master cannot hold all data in memory, the architecture must evolve to sharding with Redis Cluster.