Mastering Redis Sentinel: Monitoring, Failover, and Multi‑Sentinel Setup

What is Sentinel

Sentinel is an independent process that monitors the health of a Redis system. It has two main functions: monitoring whether the master and slave databases are running normally, and automatically promoting a slave to master when the master fails.

Monitor master and slave databases for normal operation.

Automatically convert a slave to master after a master failure.

Principle

Architecture of a single Sentinel:

Architecture of multiple Sentinels (they monitor each other as well):

Environment

The example runs in a one‑master‑multiple‑slaves setup:

Configure Sentinel

Create a Sentinel configuration file: vim sentinel.conf Add the monitoring command (replace taotaoMaster with your master name): sentinel monitor taotaoMaster 127.0.0.1 6379 1 Explanation of the parameters: taotaoMaster: name of the master to monitor (customizable, can contain letters, numbers, .-_). 127.0.0.1: IP address of the master. 6379: port of the master. 1: minimum number of votes required for a failover.

Start the Sentinel process: redis-sentinel ./sentinel.conf Sample output after starting Sentinel:

Sentinel started with ID 9059917216012421e8e89a4aa02f15b75346d2b7.

Added a monitor for the master database.

Detected two slaves (Sentinel automatically discovers slaves, no need to configure them).

Slave Crash and Recovery

After killing the slave process (PID 2826), Sentinel logs:

2989:X 05 Jun 20:09:33.509 # +sdown slave 127.0.0.1:6380 127.0.0.1 6380 @ taotaoMaster 127.0.0.1 6379

When the slave is restarted, Sentinel logs:

2989:X 05 Jun 20:13:22.716 * +reboot slave 127.0.0.1:6380 127.0.0.1 6380 @ taotaoMaster 127.0.0.1 6379
2989:X 05 Jun 20:13:22.788 # -sdown slave 127.0.0.1:6380 127.0.0.1 6380 @ taotaoMaster 127.0.0.1 6379

The slave re‑joins the replication chain, and -sdown indicates the service has recovered.

Master Crash and Recovery

When the master goes down, Sentinel prints a series of events:

2989:X 05 Jun 20:16:50.300 # +sdown master taotaoMaster 127.0.0.1 6379  说明master服务已经宕机
2989:X 05 Jun 20:16:50.300 # +odown master taotaoMaster 127.0.0.1 6379 #quorum 1/1
2989:X 05 Jun 20:16:50.300 # +new-epoch 1
2989:X 05 Jun 20:16:50.300 # +try-failover master taotaoMaster 127.0.0.1 6379  开始恢复故障
2989:X 05 Jun 20:16:50.304 # +vote-for-leader 9059917216012421e8e89a4aa02f15b75346d2b7 1  投票选举哨兵leader，现在就一个哨兵所以leader就自己
2989:X 05 Jun 20:16:50.304 # +elected-leader master taotaoMaster 127.0.0.1 6379  选中leader
2989:X 05 Jun 20:16:50.304 # +failover-state-select-slave master taotaoMaster 127.0.0.1 6379 选中其中的一个slave当做master
2989:X 05 Jun 20:16:50.357 # +selected-slave slave 127.0.0.1:6381 127.0.0.1 6381 @ taotaoMaster 127.0.0.1 6379  选中6381
2989:X 05 Jun 20:16:50.357 * +failover-state-send-slaveof-noone slave 127.0.0.1:6381 127.0.0.1 6381 @ taotaoMaster 127.0.0.1 6379  发送slaveof no one命令
2989:X 05 Jun 20:16:50.420 * +failover-state-wait-promotion slave 127.0.0.1:6381 127.0.0.1 6381 @ taotaoMaster 127.0.0.1 6379   等待升级master
2989:X 05 Jun 20:16:50.515 # +promoted-slave slave 127.0.0.1:6381 127.0.0.1 6381 @ taotaoMaster 127.0.0.1 6379  升级6381为master
2989:X 05 Jun 20:16:50.566 * +slave-reconf-done slave 127.0.0.1:6380 127.0.0.1 6380 @ taotaoMaster 127.0.0.1 6379
2989:X 05 Jun 20:16:52.438 # +switch-master taotaoMaster 127.0.0.1 6379 127.0.0.1 6381  主数据库从6379转变为6381
2989:X 05 Jun 20:16:52.438 * +slave slave 127.0.0.1:6380 127.0.0.1 6380 @ taotaoMaster 127.0.0.1 6381  添加6380为6381的从库

After the failover, instance 6381 becomes the new master with 6380 as its slave.

Configure Multiple Sentinels

To run several Sentinels, add multiple monitor lines to sentinel.conf:

sentinel monitor taotaoMaster1 127.0.0.1 6381 1
sentinel monitor taotaoMaster2 127.0.0.1 6381 2

Running multiple Sentinels provides redundancy and prevents a single point of failure in the monitoring layer.