Analysis and Fix of Redis Replication Offset Bug Causing Slave Offset Larger Than Master

In Baidu's Quality Assurance department, one of the core skills of QA is to discover, locate, and drive proper resolution of issues. This "Gold BUG" collection presents several case studies of bug discovery, analysis, and resolution.

1. Background

On 2015‑07‑31, a TOR failure in a certain network segment lasted one hour. After network recovery, multiple Redis slaves showed synchronization anomalies: all slaves except slave0 reported an offset larger than the master.

Redis uses offsets for incremental replication: a slave sends its current offset to the master, and the master sends data after that offset. An offset on the slave larger than the master indicates a flaw in the replication logic.

2. Problem Impact

Redis replication has three modes:

Full sync (first connection or incremental sync failure) – the master dumps all data to the slave.

Incremental sync (brief connection loss) – the master sends data after the slave’s reported offset.

Long‑connection sync (normal operation) – the master streams all write commands to the slave.

The issue only affects incremental sync. Two possible outcomes:

After the problem, the slave’s offset is larger than the master’s: incremental sync fails, a full sync is triggered, but the slave’s existing data remain unaffected.

After the problem, the slave’s offset is smaller than the master’s (initially larger, then the master catches up): the slave ends up with fewer data than the master.

3. Bug Localization

Abnormal Cause

The bug originates from the replica_offset handling on the slave side. The initialization and update logic are illustrated below.

Initialization: The slave receives its offset from the master after a failed incremental sync request.

When the slave receives a +FULLRESYNC response, it updates its offset.

Offset Update: After a full sync completes, each normal long‑connection sync increments the offset.

From the initialization and update flow, the abnormal offset is caused by the full‑sync process.

During the incident, the master‑slave connection is broken, so normal incremental sync does not run.

In normal sync, the slave offset can only be smaller than the master’s, never larger.

Source Code Analysis

The simplified flow of the master returning an offset is shown below:

The master first sets psync_offset to its current offset, then checks for an ongoing bgsave . If another slave is already performing bgsave , the master copies that slave’s output buffer to the new slave.

When the first slave performs a full sync, the master returns offset o1 and starts bgsave , writing new writes to repl_buffer . If a second slave requests a full sync before the first bgsave finishes, the master returns a larger offset o2 (where o2 > o1 ) and copies the first slave’s buffer to the second. After both dump phases, the second slave’s offset becomes larger than the master’s, reproducing the bug.

4. Problem Fix

After identifying the root cause, the Redis developers changed the logic: during a full sync, the master no longer returns the current offset but returns the offset saved at the start of bgsave . The same offset is cached in server.fullsync_repl_offset and returned for every full sync.

5. Testing

Post‑fix regression testing should verify the full‑sync and incremental‑sync paths. The test flow is illustrated:

If a slave requests a full sync while the master is already performing bgsave , the master marks the slave as WAIT_BGSAVE_START . After bgsave finishes, the slave’s offset may be smaller than the master’s.

6. New Problem

When the master is already in bgsave and a second slave requests a full sync, the following occurs:

The master returns offset o1 to the second slave and marks it WAIT_BGSAVE_START .

After the first bgsave ends, a new bgsave starts; the master’s offset becomes o2 and new writes are sent to the second slave’s buffer.

After dumping, the master’s offset is o2 + inc , while the slave’s offset is o1 + inc .

Since o1 < o2 , the slave’s offset is now smaller than the master’s, leading to data loss and duplicate writes on subsequent network glitches.

7. Fix Plan

The Redis upstream has provided a solution (commit bea1259 ): when a slave requests a full sync, the master no longer returns the current offset immediately but buffers it as incremental data.

8. Summary

This hidden, large‑scale bug demonstrates how subtle replication offset handling can cause serious data inconsistency. Detecting such low‑level service bugs is difficult; functional tests cannot cover every edge case, and both QA and developers need deep code understanding to create comprehensive test cases.

Testing Tips

Increase code‑review (CR) effort; understand the code to write thorough cases.

During CR, draw flowcharts of the logic, check variable initialization, memory allocation/release, and branch correctness.

Bug‑Localization Tips

Identify the exact variable or function causing the issue.

Read the code to locate where the variable/function is initialized and modified.

Analyze possible error reasons and verify the corresponding logic.