Redis Sentinel Deep Dive: Leader Election, Failover Mechanics, and Production Best Practices

Why Sentinel Exists

Redis Sentinel does not guarantee "Redis never fails". It answers three concrete questions: who decides a master is unavailable, who is authorized to perform the failover, and how clients discover the new master.

Real‑World Incident

An e‑commerce product‑detail service caches product info, marketing tags and short‑lived inventory values in a Redis cluster with one master, two replicas and three Sentinel instances. During a midnight traffic spike the master host lost network connectivity. Sentinel eventually promoted a replica, but the business still saw 5xx errors because: down-after-milliseconds was set to 30 s, making detection too slow.

One Sentinel ran on the same node as the master; after the node failed only two Sentinels remained and could not form a majority during network jitter.

Some clients connected directly to the old master IP instead of using Sentinel‑aware discovery.

All 60 pods rebuilt connections simultaneously, creating a connection storm.

Cache misses were forwarded without limits to PostgreSQL, causing a database snowball.

Sentinel only handles Redis topology failover; it does not provide business‑level degradation, database protection, command idempotency or automatic client IP replacement.

Fault‑Budget Decomposition

The end‑to‑end outage window consists of:

Business outage ≈ failure detection + ODOWN negotiation + leader election + replica promotion + config propagation + client discovery/reconnection

Each stage is affected by network RTT, event‑loop latency, number of alive Sentinels, replica freshness and client behaviour. SLOs should be derived from fault‑injection drills rather than static configuration values.

Sentinel Responsibilities (Four Duties)

Monitoring : Periodically ping master, replicas and other Sentinels.

Notification : Emit logs, Pub/Sub messages or trigger scripts on state changes.

Automatic Failover : Elect an executor, promote a suitable replica and reconfigure the remaining replicas.

Configuration Provider : Clients query the master name to obtain the current master address.

The control plane (Sentinel) decides the new topology, the data plane (Redis) stores the data, and the client layer performs the traffic switch.

What Sentinel Does Not Do

No sharding – a single master group is limited by one thread and node resources.

No strong consistency – replication is asynchronous, so a short window of data loss exists.

No automatic client repair – direct IP connections, stale DNS caches or infinite retries remain the client’s responsibility.

No cross‑region disaster recovery – latency, partitioning and replication lag must be handled by a separate DR design.

SDOWN vs ODOWN

When a Sentinel (e.g., S1) does not receive a valid reply within down-after-milliseconds, it marks the master as SDOWN (subjectively down). SDOWN is a local opinion; other Sentinels may disagree.

S1 then asks the rest of the Sentinels via SENTINEL is-master-down-by-addr. If a configurable quorum of Sentinels also consider the master down, the master becomes ODOWN (objectively down). Only ODOWN triggers failover, and a separate majority of Sentinels must authorize the actual switch.

Quorum is not the same as majority: quorum decides when ODOWN is set, while majority decides which Sentinel may execute the failover. The effective threshold is max(quorum, floor(totalSentinels/2)+1).

Raft‑like Leader Election

Candidate Sentinel increments its current epoch.

It sends SENTINEL is-master-down-by-addr to other Sentinels for a vote in that epoch.

Each Sentinel may vote only once per epoch; the first valid request usually wins.

The candidate that gathers the required majority becomes the leader for this round.

If no candidate wins, a new epoch is started and the process repeats.

The algorithm resembles Raft’s term‑based single‑vote approach but is not a full Raft log replication system.

Replica Selection Algorithm

After a leader is elected, Sentinel chooses the new master in two phases.

Phase 1 – Filtering

Exclude replicas that are SDOWN.

Exclude replicas that cannot be reached or cannot return INFO.

Exclude replicas with replica-priority=0.

Exclude replicas that have been disconnected from the old master for too long. The threshold is

(down-after-milliseconds × 10) + time‑master‑was‑SDOWN

.

Phase 2 – Sorting

Lower replica-priority wins.

If priorities tie, the higher replication offset wins.

If offsets also tie, the lexicographically smaller run ID wins (deterministic tie‑breaker).

Example:

Replica A: priority=50, offset=9,991,000, runID=b91...

Replica B: priority=100, offset=9,999,900, runID=a12...

Replica C: priority=0, offset=10,000,000, runID=c33...

Result: Replica A wins because its priority (50) is lower than B’s (100) even though B’s offset is higher; C is never eligible because priority 0 disables promotion.

Failover State Machine (Six Stages)

WAIT_START : Leader waits for authorization.

SELECT_SLAVE : Filters and selects the best replica.

SEND_SLAVEOF_NOONE : Issues REPLICAOF NO ONE on the chosen replica to make it a master.

WAIT_PROMOTION : Confirms the new role via INFO.

RECONF_SLAVES : Reconfigures remaining replicas to follow the new master (parallel‑syncs controls concurrency).

UPDATE_CONFIG : Publishes the new configuration so clients can discover the new master.

After the new master is confirmed, the failover is considered successful, although other replicas may still be synchronising.

Real‑World Parameter Tuning

down-after-milliseconds

: Too low → false positives; too high → slow detection. failover-timeout: Too low → repeated failures; too high → slow recovery. parallel-syncs: Controls how many replicas re‑sync concurrently; high values can saturate network, low values delay full recovery. repl-backlog-size: Must cover peak replication throughput × expected outage duration × safety factor. min-replicas-to-write and min-replicas-max-lag: Provide best‑effort write protection at the cost of availability.

Sample Redis Configuration

# redis.conf
port 6379
protected-mode yes
appendonly yes
appendfsync everysec
aof-use-rdb-preamble yes
min-replicas-to-write 1
min-replicas-max-lag 5
repl-backlog-size 256mb
repl-backlog-ttl 3600
replica-priority 100
replica-read-only yes
client-output-buffer-limit replica 512mb 128mb 60
lazyfree-lazy-eviction yes
lazyfree-lazy-expire yes

Sample Sentinel Configuration

# sentinel.conf
port 26379
protected-mode yes
sentinel monitor orders-cache redis-0.redis-headless.cache.svc.cluster.local 6379 2
sentinel down-after-milliseconds orders-cache 5000
sentinel failover-timeout orders-cache 60000
sentinel parallel-syncs orders-cache 1
sentinel auth-user orders-cache sentinel-replication
sentinel auth-pass orders-cache ${SENTINEL_REDIS_PASSWORD}
sentinel resolve-hostnames yes
sentinel announce-hostnames yes

Important notes: down-after-milliseconds=5000 must exceed normal P99.9 latency, GC pauses and acceptable event‑loop blocking. parallel-syncs=1 reduces simultaneous replica loading, protecting read capacity.

Sentinel rewrites its configuration at runtime; the config file must be writable (e.g., mounted from a writable volume, not a read‑only ConfigMap).

Kubernetes Deployment Patterns

StatefulSet provides stable pod names and persistent volumes, but pod names (e.g., redis‑0) do not permanently map to master or replica roles. After a failover any pod can become master.

Headless Service supplies stable DNS entries for each pod. publishNotReadyAddresses can be enabled to allow discovery before pods are Ready, but clients must still query Sentinel for the master address.

PodDisruptionBudget should guarantee at least two Sentinels remain available (e.g., minAvailable: 2 for a three‑Sentinel deployment).

Topology spread constraints or anti‑affinity rules should distribute Redis pods and Sentinels across nodes and AZs to avoid a single point of failure.

Spring Boot Integration (Java 21, Spring Boot 3.x, Lettuce)

Two‑level cache: an in‑process Caffeine L1 cache (TTL ≈ 3 s) and a Redis L2 cache (TTL ≈ 10 m with jitter). A semaphore limits concurrent database fall‑backs to 32, protecting the DB connection pool.

package com.acme.catalog.infrastructure;

import io.lettuce.core.ReadFrom;
import org.springframework.boot.autoconfigure.data.redis.LettuceClientConfigurationBuilderCustomizer;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

@Configuration
public class RedisClientConfig {
    @Bean
    LettuceClientConfigurationBuilderCustomizer masterReadOnly() {
        return builder -> builder.readFrom(ReadFrom.MASTER);
    }
}

Key service methods (simplified):

public ProductView get(long productId) {
    return localCache.get(productId, this::loadFromRedisOrDatabase);
}

private ProductView loadFromRedisOrDatabase(long productId) {
    String key = KEY_PREFIX + productId;
    try {
        String json = redis.opsForValue().get(key);
        if (json != null) {
            return objectMapper.readValue(json, ProductView.class);
        }
    } catch (DataAccessException | JsonProcessingException e) {
        redisFailure.increment();
        log.warn("redis read degraded, productId={}", productId, e);
    }
    if (!dbBulkhead.tryAcquire()) {
        dbRejected.increment();
        throw new CacheOriginOverloadedException("database fallback is saturated");
    }
    try {
        ProductView product = repository.findPublishedById(productId)
            .orElseThrow(() -> new NoSuchElementException("product not found: " + productId));
        writeRedisBestEffort(key, product);
        return product;
    } finally {
        dbBulkhead.release();
    }
}

private void writeRedisBestEffort(String key, ProductView product) {
    try {
        long jitterSeconds = ThreadLocalRandom.current().nextLong(0, 61);
        redis.opsForValue().set(key, objectMapper.writeValueAsString(product),
            redisTttl.plusSeconds(jitterSeconds));
    } catch (DataAccessException | JsonProcessingException e) {
        redisFailure.increment();
        log.warn("redis refill failed, productId={}", product.productId(), e);
    }
}

Cache eviction uses UNLINK for asynchronous memory reclamation; failures fall back to TTL expiry.

High‑Concurrency Guardrails

Connection‑storm mitigation: share client connections, set connection and command timeouts, use exponential back‑off with jitter, cap maxclients on Redis.

Cache‑breakdown protection: L1 short‑TTL buffer, request coalescing (Caffeine), global back‑end semaphore for DB fallback.

Resource isolation: separate thread pools for Redis I/O and DB fallback; align DB pool size with semaphore permits.

Scaling signals: memory approaching limit, CPU/network saturation, write throughput exceeding single‑master capacity – trigger migration to Redis Cluster.

Observability Stack

Sentinel Metrics & Events

CKQUORUM success/failure.

Known Sentinel count, healthy replica count.

Flags s_down, o_down.

Event stream: +sdown, +odown, +try-failover, +elected-leader, +selected-slave, +promoted-slave, +switch-master, -odown.

Failover duration and rate per time window.

Redis Data‑Plane Metrics

connected_slaves

, master_repl_offset, per‑replica offset/lag. master_sync_in_progress, sync_partial_ok/err, sync_full.

Ops/sec, network I/O, connection count.

Memory usage, fragmentation, evicted keys.

Fork latency, AOF rewrite/RDB status.

Command latency, slowlog, event‑loop latency.

Client‑Side Metrics

Command success/timeout rates (P50/P95/P99).

Master address changes observed by the client.

Reconnection attempts and latency.

L1/L2 hit rates, DB fallback volume, fallback rejection count.

503 degradation rate, overall request latency, error‑budget consumption.

Alerting should correlate Sentinel ODOWN events with rising Redis timeouts and DB fallback spikes to prioritize incidents that truly impact the business.

Logging & Security

Logs may include master name, node address, epoch, failover duration and request trace IDs, but must never expose passwords, ACL tokens, full cache values or user‑sensitive data. Sentinel and Redis should use separate authentication accounts, and secret rotation must be exercised end‑to‑end.

Fault‑Injection & Playbooks

Pre‑Launch Checks

# Verify quorum and majority
redis-cli -h sentinel-0 -p 26379 SENTINEL CKQUORUM orders-cache
# Confirm current master address
redis-cli -h sentinel-0 -p 26379 SENTINEL GET-MASTER-ADDR-BY-NAME orders-cache
# List replicas and their priorities
redis-cli -h sentinel-0 -p 26379 SENTINEL REPLICAS orders-cache
# Check replication health on the master
redis-cli -h current-master -p 6379 INFO replication
# Ensure Sentinel config is writable
redis-cli -h sentinel-0 -p 26379 SENTINEL FLUSHCONFIG

Five‑Category Failure Matrix

Master process exit : graceful Redis stop – verify detection → election → promotion → client recovery.

Master node crash : power‑off or kill host – validate Sentinel‑replica co‑location and that failover proceeds.

Network partition : NetworkPolicy or chaos‑mesh – minority should not trigger failover; old master must be write‑blocked.

Replica lag : throttle bandwidth or pause replica – confirm leader avoids stale replica during promotion.

Sentinel loss : stop one Sentinel, then a second – check CKQUORUM, alerts and inability to reach majority.

Never use SENTINEL FAILOVER in production for full‑stack testing; it bypasses real failure detection and only validates the switch path.

Acceptance Metrics

T0  fault injection
T1  first Sentinel marks SDOWN
T2  ODOWN achieved
T3  Leader elected
T4  Replica promoted
T5  First successful client command to new master
T6  All replicas healthy

Measure T5‑T0 for business recovery time and T6‑T0 for topology stabilization.

Automated Integration Test Sketch

Write a versioned product cache entry.

Query Sentinels for the current master.

Stop the master or isolate its network.

Poll multiple Sentinels until they agree on a new master.

Use the application client to GET the product (should succeed via new master).

Restart the old master and verify it becomes a replica.

Assert that DB fallback concurrency never exceeds the configured limit.

On failure, capture Sentinel logs, INFO replication output and client connection traces.

Operational Checklist Before Going Live

Deploy at least three Sentinels across independent nodes/AZs.

Validate quorum and majority with CKQUORUM.

Ensure at least two healthy replicas with monitored replication lag.

Set appropriate replica‑priority on all candidates.

Make sentinel.conf writable and persisted across restarts.

Applications must use Sentinel‑aware clients and a stable master name.

Separate ACL/TLS credentials for data nodes and Sentinels.

Define command timeouts; retry only for safe read operations.

Implement L1 cache, DB fallback bulkhead, and TTL jitter.

Confirm business‑level impact of min‑replicas‑* settings.

Verify K8s anti‑affinity, PodDisruptionBudget, resource limits and NetworkPolicy.

Complete failure‑injection drills for master crash, network split, replica lag and Sentinel loss.

Instrument Sentinel, Redis, client and DB layers with unified metrics.

Define clear capacity thresholds for migrating to Redis Cluster or a managed service.

Core Takeaways

Two‑layer failure detection : SDOWN is a local opinion; ODOWN requires quorum.

ODOWN ≠ automatic failover : A separate majority must authorize the switch.

Leader election filters then sorts : health → priority → offset → run‑ID.

Failover is a state machine : promotion, replica reconfiguration, config propagation, client reconnection.

High availability is end‑to‑end : Sentinel handles control‑plane failover, but data consistency, cache degradation, DB protection, idempotency and operational playbooks must be engineered separately.

When a single‑master capacity meets the workload, data can be rebuilt from the source database and Sentinel offers a mature HA solution. If write throughput, data size or cross‑region consistency exceed a single master’s limits, migrate to Redis Cluster, a managed service or a different data system rather than over‑loading Sentinel.