How to Build a Reliable Redis Sentinel HA Setup and Fix Password Auto‑Rewrite

Overview

Redis is widely used as a cache or high‑performance store, so its availability is critical. The dominant high‑availability (HA) design combines Redis Sentinel with master‑slave replication. Sentinel, the official HA middleware, works reliably with Redis 3.0 and newer.

HA Architecture and Deployment

The recommended topology consists of:

One Redis master.

Three Redis replicas (slave nodes) replicating from the master.

Three independent Sentinel instances, each running on a different physical server.

Each component is installed and started according to the standard Redis and Sentinel documentation; the steps are omitted for brevity.

Application Configuration

Clients should connect to Sentinel rather than directly to the master. The typical configuration includes:

Connection strings for the three Sentinel nodes (e.g., sentinel1:26379,sentinel2:26379,sentinel3:26379).

The Redis password used for AUTH.

The masterName as known to Sentinel (default is mymaster).

If Sentinel is fronted by a load‑balancer such as LVS, replace the three addresses with the virtual IP (VIP) of the load‑balancer.

At runtime the client queries Sentinel for the current master address, then performs all reads and writes against that master. Sentinel adds only negligible latency.

Best‑Practice Considerations

When multiple Sentinel endpoints are supplied, drivers should select one using round‑robin or random selection to distribute load.

If a Sentinel node becomes unreachable, the driver must retry other nodes, optionally maintaining a temporary blacklist of failed endpoints.

Use drivers that provide connection‑pooling so that TCP connections can be reused across requests, reducing connected_clients counts.

Problem Analysis

During certain events—master failover, a new Sentinel joining the cluster, execution of CONFIG REWRITE, SENTINEL FLUSHCONFIG, SENTINEL REMOVE, or addition of a new master—Redis rewrites the password configuration by surrounding the value with double quotes (e.g., "myPass"). Because passwords containing literal quotes are prohibited, this automatic quoting can break authentication and increase operational risk.

Remediation Options

Manual correction : Monitor the password value and manually edit it back after an alert. Simple but prone to human error.

Automated script : Deploy a periodic script that detects a changed password and restores the original value. This adds operational overhead and cannot guarantee 100 % reliability.

Source‑code modification : Patch Redis source to stop rewriting the password fields (see next section).

Source‑code Modification

In the Redis source tree, edit src/config.c. Locate the function int rewriteConfig(char *path) and comment out the two lines that rewrite the masterauth and requirepass options:

// rewriteConfigStringOption(state,"masterauth",server.masterauth,NULL);
// rewriteConfigStringOption(state,"requirepass",server.requirepass,NULL);

Recompile Redis (e.g., make && make install) and restart the server. Verify that CONFIG REWRITE no longer adds quotes to the password. The author reports stable operation for over a year after applying this patch.

Resources

Patched Redis source code repository:

https://github.com/giantmangu888/redis3.0.7

Clone command:

git clone https://github.com/giantmangu888/redis3.0.7.git