Nacos CP Mode Still Has Split-Brain? 4 Scenarios Where Raft Fails

Counter-Intuitive Fact: CP Mode Does Not Guarantee Zero Split-Brain

Many assume switching Nacos to CP mode (Raft protocol) eliminates split-brain. This is false. CP only replaces "temporary inconsistency allowed" with "majority acknowledgment required." Under specific network conditions — GC pauses, cross-AZ jitter, even-node partitions — Raft can still elect multiple leaders (causing data conflicts) or elect no leader (causing write outage).

What Nacos CP Mode Actually Does

Nacos 2.x defaults to Distro protocol (AP mode) with periodic full sync via DataSyncTask. Enabling CP mode requires:

spring:
  cloud:
    nacos:
      discovery:
        naming-load-cache-urgently: true  # temporary CP switch

Under the hood, DistroConsistencyServiceImpl implements Raft: write requests go to leader → log replicated to majority → commit → respond. Key constraint: writes need majority confirmation (3 nodes need 2, 5 nodes need 3). But this assumes nodes communicate normally; once communication breaks, the guarantee collapses.

Four Split-Brain Trigger Scenarios in CP Mode

Scenario 1: Even-Node Fatal Partition

A 4-node cluster splits 2+2. Neither partition has majority (needs 3). Result: both partitions fail to elect a leader, cluster enters "cannot write" deadlock, all registration requests time out. This is not dual-leader split-brain but a no-leader paralysis — more hidden, more fatal.

Scenario 2: GC Pause "Fake-Death" Election

JVM Full GC lasting seconds causes heartbeat timeout; other nodes mark it dead and start election. Mid-election, GC ends and node rejoins voting. Possible outcomes:

New leader elected but old node rejects it

Two nodes believe they can handle writes

Recommended JVM params:

export JAVA_OPT="${JAVA_OPT} -Dnacos.member.raft.rpc.timeout=2000"
export JAVA_OPT="${JAVA_OPT} -Dnacos.member.raft.election.timeout=5000"

If GC pause exceeds election.timeout, repeated elections create an "election storm."

Scenario 3: Cross-AZ Network Jitter

Nodes deployed across AZs; brief inter-AZ disruption (high latency + packet loss, not full outage). Heartbeats sent but ACKs lost → nodes misjudged as down → both sides attempt election. Default heartbeat interval 5000ms, timeout 15000ms — too aggressive for cross-region or public network.

Scenario 4: Snapshot Load Failure Causing "History Fork"

Node restarts and restores from snapshot. If snapshot captured while uncommitted logs existed, restored node lags behind cluster but doesn't know. It then votes and accepts writes, creating data divergence.

Why CP Mode Still "Looks Like AP"

4.1 Client Local Cache Unconstrained by CP

Nacos Client defaults naming.load.cache.enable=true, loads full instance cache at startup, TTL 30s. Server may be CP-consistent, but client holds 30-second-old "stale snapshot" → consumer sees instance list mismatching actual server state. Server consistency ≠ client consistency. CP only governs server side.

4.2 Read Requests Bypass Raft

Raft constrains writes only. Reads ( getInstances) hitting a follower may return stale data. CP mode defaults reads to leader, but if client connects directly to a follower, consistency is lost.

4.3 Metadata and Service Data Stored Separately

Nacos stores service registration and config data separately. CP primarily protects config data (Config module); naming module may still use Distro protocol in some versions. You think full CP switch, but only half is CP. Verify:

curl -X GET "http://nacos-server:8848/nacos/v1/ns/operator/servers"
# Check each node's role field; expect exactly one LEADER

Practical Steps to Make CP Mode Truly Effective

5.1 Node Count Must Be Odd, Minimum 3

conf/cluster.conf  # must list all nodes
192.168.1.100:8848
192.168.1.101:8848
192.168.1.102:8848

3 nodes tolerate 1 failure, 5 tolerate 2. Even nodes guarantee no-leader deadlock during partition.

5.2 Heartbeat and Timeout Tuning

application.properties (Nacos Server)
nacos.core.cluster.heartbeat.interval=5000        # heartbeat interval 5s
nacos.core.cluster.heartbeat.timeout=15000        # timeout 15s (3x interval)
nacos.core.cluster.election.timeout=5000          # election timeout 5s
nacos.core.cluster.heartbeat.failure.threshold=3  # 3 consecutive failures before marking down
nacos.core.cluster.communication.timeout=5000     # inter-node communication timeout

For cross-AZ, increase heartbeat.timeout to 30000ms+ to avoid false failure detection.

5.3 Force Disable Client Cold Load

spring:
  cloud:
    nacos:
      discovery:
        naming-load-cache-at-start: false  # no cache at startup
        ephemeral: false                   # persistent instances, avoid accidental deletion
        fail-fast: true                    # fast fail, reject stale data

5.4 Enable Server Push + Client Incremental Listening

# Nacos Server
naming.push.receiver.enable=true

# Client: use UDP push instead of polling
# Client processes only incremental changes, not full pulls

5.5 Deployment Architecture: Cross-AZ + Unitization

AZ-A: Nacos node1, node2
AZ-B: Nacos node3
AZ-C: Nacos node4, node5

Client: prefer same-AZ node, cross-AZ fallback

Don't put all eggs in one AZ. Single AZ failure leaves other AZ nodes to form majority.

Diagnostic Toolchain

When suspecting split-brain, execute in order:

# 1. Check cluster node roles
curl -s "http://nacos-server:8848/nacos/v1/ns/operator/servers" | jq '.role'
# Expect exactly one "LEADER", rest "FOLLOWER"
# Multiple LEADERs → confirmed split-brain

# 2. Check election logs
grep "election" /nacos/logs/naming-raft.log | tail -50
# Frequent entries → election storm, network unstable

# 3. Compare instance registration counts per node
curl -s "http://nacos-server:8848/nacos/v1/ns/instance/list?serviceName=your-service" | jq '.hosts | length'
# Run on each node, compare results

# 4. Use nacos-checker for data consistency
java -jar nacos-checker.jar --server http://nacos-server:8848

Overlooked Truth: Root Cause Often in Infrastructure

Network device (switch, firewall) rule changes intercepting heartbeats

Kubernetes pod rescheduling changes node IPs, cluster config not updated

Cloud provider security group changes not synced to inter-node communication ports

Nacos is just the tip of the distributed systems iceberg; the entire network infrastructure underpins it. Unstable infrastructure makes perfect Raft protocol meaningless.

Summary: CP Is a Conditional Contract

Odd nodes ≥ 3 : When met, majority achievable; when violated, even nodes partition → no leader.

Network stable RTT < 50ms : When met, heartbeats normal, elections stable; when violated, high latency → false down → election storm.

JVM GC pause < election timeout : When met, node not misjudged; when violated, long GC → fake death → dual leader.

Client cache disabled / TTL reasonable : When met, client view ≈ server view; when violated, cold load + long TTL → client sees stale data.

Naming module truly runs CP : When met, service registration strong consistency; when violated, module mix → half CP half AP.

CP mode is a contract with preconditions. True high availability requires all five: odd nodes + network redundancy + parameter tuning + client governance + solid infrastructure — none can be missing.