Understanding Redis Sentinel: Architecture, Configuration, and Automatic Failover

1. Redis Sentinel Overview

Redis Sentinel provides automatic high‑availability for Redis by monitoring master and replica instances, detecting failures, and performing failover without manual intervention.

2. Sentinel Architecture

Sentinel consists of two types of nodes:

Sentinel nodes : special Redis instances that do not store data but monitor the cluster.

Data nodes : the actual master and replica servers.

Sentinel adds automated fault‑recovery on top of Redis replication.

3. Core Sentinel Functions

Monitoring : periodic PING commands to masters and replicas; unresponsive nodes are marked down.

Notification : when a problem is detected, Sentinel notifies administrators or applications via API and sends REPLICAOF commands to reconfigure replicas.

Automatic failover : promotes a suitable replica to master and re‑links other replicas.

Configuration provider : clients query Sentinel to obtain the current master address.

4. Hello World – Deploying Master/Slave and Sentinel Nodes

4.1 Master‑Slave Configuration

Typical redis.conf files:

#redis.conf  master
port 6379
daemonize yes
logfile "6379.log"
dbfilename "dump-6379.rdb"

#redis_6380.conf
port 6380
daemonize yes
logfile "6380.log"
dbfilename "dump-6380.rdb"
replicaof 127.0.0.1 6379

#redis_6381.conf
port 6381
daemonize yes
logfile "6381.log"
dbfilename "dump-6381.rdb"
replicaof 127.0.0.1 6379

Start the instances with:

redis-server redis.conf
redis-server redis_6380.conf
redis-server redis_6381.conf

Verify replication using redis-cli -p 6379 info replication.

4.2 Sentinel Configuration

Three sentinel config files (only the port and monitor line differ):

# redis-sentinel-26379.conf
port 26379
daemonize yes
logfile "26379.log"
sentinel monitor mymaster 127.0.0.1 6379 2

# redis-sentinel-26380.conf
port 26380
daemonize yes
logfile "26380.log"
sentinel monitor mymaster 127.0.0.1 6379 2

# redis-sentinel-26381.conf
port 26381
daemonize yes
logfile "26381.log"
sentinel monitor mymaster 127.0.0.1 6379 2

Start them with redis-sentinel redis-sentinel-26379.conf (similarly for the other two).

4.3 Checking Sentinel Status

Connect to a sentinel and run info sentinel to see the monitored master and the number of sentinels.

5. Demonstrating Failover

Kill the master process (port 6379). Initially, sentinels still report the old master; after a short interval they promote one replica (e.g., port 6381) to master and update their state.

During the transition, the former master is marked as a replica of the new master, and the cluster’s epoch counter is incremented.

6. Sentinel Mechanism Details

6.1 Subjective vs. Objective Down

Sentinel first marks a node “subjectively down” after a missed PING. Only when a majority of sentinels agree is the node marked “objectively down”, triggering failover.

6.2 Leader Election

Sentinels use the Raft algorithm to elect a leader sentinel that coordinates the failover.

6.3 Failover Steps

Select a healthy replica (filtering out those flagged subjectively down or with stale ping responses).

Promote it with SLAVEOF NO ONE and reconfigure other replicas to follow it using SLAVEOF.

Convert the old master into a replica of the new master.

6.4 Replica Selection Scoring

Sentinel scores candidates by:

Replica priority ( slave-priority).

Replication offset closeness to the master.

Lowest run‑id when priority and offset are equal.

7. Sentinel Cluster Principles

A sentinel cluster is formed when three or more sentinel instances exchange state via a Pub/Sub channel __sentinel__:hello. They also periodically:

Send INFO to masters to discover replicas.

Publish their own status to the hello channel every 2 seconds.

Ping all known nodes every second for liveness checks.

8. Pub/Sub‑Based Cluster Formation

Sentinels publish their IP/port on the hello channel; other sentinels subscribe to discover each other and build the cluster.

9. Connections to Replicas and Clients

Sentinels obtain replica addresses via INFO from the master, then open connections to monitor them. After failover, they notify clients by publishing events (e.g., +odown) that clients can subscribe to.

10. Important Sentinel Commands

Command

Purpose

PING

Sentinel checks the liveness of a Redis node.

INFO

Sentinel retrieves replica information from the master.

PUBLISH

Sentinel publishes its own info and master configuration to __sentinel__:hello.

SUBSCRIBE

Sentinel subscribes to the hello channel to learn about other sentinels.

SENTINEL IS-MASTER-DOWN-BY-ADDR

Used among sentinels to agree on a master’s down state and start election.

11. Recommendations

Deploy sentinel nodes on separate physical machines.

Use an odd number of sentinels (≥ 3) to avoid split‑brain scenarios.

12. References

《Redis 开发与运维》

《Redis 核心技术与实战》

https://redis.io/topics/sentinel

https://www.cnblogs.com/kismetv/p/9609938.html