Mastering Raft Member Changes: Joint Consensus vs Single‑Step Strategies

Introduction

Member changes are an unavoidable difficulty in implementing consistency systems, and they greatly improve operational capability and service availability.

This article starts from Raft member‑change theory, introduces Raft member changes and single‑step member changes, including the well‑known Raft bug, and provides engineering practice experience.

Raft Member‑Change Overview

During the operation of a distributed system, nodes may fail and need to be dynamically added or removed. Member change modifies the set of nodes participating in the consensus protocol (adding, removing, or replacing nodes) without affecting system availability.

Member change is also a consistency problem: all nodes must agree on the new configuration. However, during the change the voting set itself changes, which can lead to two disjoint majorities and potentially two leaders, breaking safety.

When expanding a 3‑node cluster to 5 nodes, a naïve expansion can create two separate majorities (old and new), leading to conflicting decisions.

To avoid this, Raft proposes a two‑phase method called Joint Consensus.

1. Joint Consensus Member Change

Joint Consensus first switches the cluster from the old configuration to a transitional configuration that combines the old and new members (Joint Consensus configuration). Once the joint configuration is committed, the cluster switches to the new configuration.

The leader first replicates a joint‑configuration entry that must be confirmed by both old and new majorities. After it is committed, the leader replicates a second entry containing only the new configuration, which requires confirmation only from the new majority.

2. Single‑Step Member Change

Single‑step changes restrict each change to adding or removing a single member, ensuring that any two majorities always intersect. This allows a direct switch from the old configuration to the new one without a joint phase.

Although the theory is simple, practical implementations encounter many pitfalls.

Problems with Raft Single‑Step Member Change

1. Correctness Issue

If a leader fails during a single‑step change, the new leader may commit logs using a different configuration, causing the old leader’s uncommitted member‑change entry to be overwritten and resulting in data loss.

Example: a 4‑node cluster adds two new nodes; a leader switch occurs before the member‑change entry is committed, leading to lost logs.

The root cause is that the previous leader’s member‑change log was not replicated to a majority before it crashed. The new leader may form a different majority, causing a split‑brain and overwriting committed logs.

The Raft authors propose a fix: the new leader must first commit a no‑op entry before replicating any member‑change entry, ensuring at least one node overlaps with the previous configuration.

2. Availability Issue

Single‑step changes require two separate operations for member replacement (add then remove). If a network partition occurs between the two steps, the service may become unavailable.

Example: three members in different data centers need to replace one member. The process "add new → remove old" may be impossible under certain partition scenarios.

Engineering Practices for Raft Member Changes

1. When to Add a New Member and Sync Data

Two options exist:

Add first, then sync data: Simple and fast, but the new member cannot serve until data is synced, and the enlarged majority may reduce availability during failover.

Sync data first, then add: The new member joins as a learner without voting rights, data is synced, then it becomes a full member; this preserves availability but is more complex and cannot add a member that does not yet exist.

2. Configuration Used by Member‑Change Logs

For Joint Consensus, the log that adds the joint configuration requires confirmation from both old and new majorities; the log that finalizes the new configuration requires only the new majority.

3. When Member‑Change Logs Take Effect

Joint Consensus logs become effective on the leader before the log is sent, and on followers after the log is persisted.

4. Ordering of Logs During Member Change

Even though the majority size may shrink after a change, Raft’s commit algorithm ensures that any log that achieves a majority is safely committed, so ordering does not break safety.

5. Recovering When Only a Few Members Survive

If a majority cannot be formed, a forced configuration change interface can be used to set a new member list (e.g., a single surviving node becomes the sole member) to restore service.

Single‑Step Member Change Engineering Practices

1. Which Configuration to Use for the Log

Using the old configuration avoids the correctness issue but may require a larger majority when removing members.

Using the new configuration requires the no‑op fix but may need a larger majority when adding members.

2. When the Log Takes Effect

If the new configuration is used, the log becomes effective before the leader starts replication and after followers persist it, similar to Joint Consensus. If the old configuration is used, it typically becomes effective after the log is committed.

Conclusion

Raft offers both Joint Consensus and single‑step member changes. Joint Consensus is generally safer and easier to reason about, while single‑step changes are prone to correctness and availability pitfalls. In practice, it is recommended to prefer Joint Consensus for cluster reconfiguration.