Designing Redis High Availability with Sentinel

Redis Sentinel

Sentinel is the official high‑availability solution for Redis clusters. It can automatically perform failover, reducing manual intervention, and provides clients with monitoring notifications so they can adapt to server state changes.

Key functions of Sentinel include:

Monitoring – continuously checks the health of masters and slaves.

Notification – alerts administrators or scripts via API when an instance is down.

Automatic failover – promotes a slave to master when the current master fails and reconfigures other slaves.

Configuration provider – clients query Sentinel for the current master address and receive updates after a failover.

Sentinel Configuration

Sentinel runs as a special‑mode Redis server; its behavior is defined in sentinel.conf :

// [monitor name] [ip] [port] [number of sentinels that must agree to trigger failover]
sentinel monitor mymaster 127.0.0.1 6379 2
// [down‑after‑milliseconds] – time after which a master is considered subjectively down
sentinel down-after-milliseconds mymaster 60000
// [failover timeout]
sentinel failover-timeout mymaster 180000
// [parallel‑syncs] – max number of slaves that can sync with the new master simultaneously
sentinel parallel-syncs mymaster 1

Sentinel requires Redis version 2.8 or higher. Start it with:

redis-sentinel sentinel.conf

After startup, Sentinel performs:

Every 10 seconds, sends an INFO command to the monitored master to retrieve its status.

Every 1 second, PINGs all Redis servers (including other Sentinels) to check liveness.

Every 2 seconds, publishes the current Sentinel and master information to all monitored nodes.

It is recommended to run at least three Sentinel instances and configure the quorum accordingly (e.g., 2 out of 3 for a failover).

Three Ways to Receive Failover Notifications

1. Script Reception

During a failover you can specify a notification script that informs administrators of the cluster state.

The script may run for a maximum of 60 seconds; exceeding this limit causes it to be killed.

sentinel notification-script mymaster /var/redis/notify.sh

After failover, you can configure a client‑reconfiguration script.

sentinel client-reconfig-script mymaster /var/redis/notifyReconfig.sh

2. Direct Client Reception

Sentinel publishes failover events via Redis Pub/Sub. Clients simply subscribe to the relevant channels (e.g., "+slave") to receive messages.

Message format:

@

Example notification:

*          // subscription type, * means all events
-sdown      // message type
slave 127.0.0.1:6379 127.0.0.1 6379 @ mymaster 127.0.0.1 6381

Sample subscription code (C# style):

using (RedisSentinel rs = new RedisSentinel(CurrentNode.Host, CurrentNode.Port))
{
    var redisPubSub = new RedisPubSub(node.Host, node.Port);
    redisPubSub.OnMessage += OnMessage;
    redisPubSub.OnSuccess += (msg) => {};
    redisPubSub.OnUnSubscribe += (obj) => {};
    redisPubSub.OnError = (exception) => {};
    redisPubSub.PSubscribe("*");
}

3. Indirect Service Reception

Instead of having the application subscribe directly, a separate service can listen to Sentinel events and expose an API for the application to receive callbacks. Benefits include reduced coupling, lower chance of listener failures, and easier extensibility.

If Sentinel is later replaced, only the service needs to change.

The service can add a watchdog thread to poll Redis status, ensuring detection even if Sentinel fails (though this is a rare edge case).

Example workflow:

http://127.0.0.1/redis/notify.api

The service, upon detecting a state change, calls the application’s API:

httprequest.post("http://127.0.0/redis/notify.api");

Overall Design Recommendation

The third method (indirect service) is recommended for production environments. The overall flow diagram is shown below:

Summary

Sentinel supports various notification types documented at the official Redis site. Example client implementations are available on GitHub (https://github.com/mushroomsir/HRedis). While Zookeeper can be used for HA, Sentinel offers a lightweight, cost‑effective solution suitable for small‑to‑medium deployments.

There is no single “best” architecture; the goal is to choose the most suitable one for your constraints.

Reference: http://redis.io/topics/sentinel