Dissecting Nacos 3.x Distro Protocol: The Distributed Design Behind the 1000 ms Delay

Why Nacos Built Distro Instead of Using Gossip

Nacos already uses JRaft for CP consistency. For the AP side it could have adopted a gossip protocol, but the Alibaba team designed Distro because gossip “talks to everyone”, while Distro “lets each node handle only its own slice and then informs others”. In service‑registration scenarios only the node responsible for a client needs to propagate the change, reducing message volume by roughly an order of magnitude.

Overall Architecture – Three‑Layer Model

Distro’s source code lives in two modules, core and naming. The core module defines DistroProtocol with four entry methods:

public void sync(DistroKey distroKey, DataOperation action) { … }
public void syncToTarget(DistroKey distroKey, DataOperation action, String targetServer, long delay) { … }
public boolean onReceive(DistroData distroData) { … }
public boolean onVerify(DistroData distroData, String sourceAddress) { … }

DistroProtocol

itself stores no data and sends no requests; it is a routing layer that selects a TransportAgent, DataStorage and DataProcessor based on resourceType and then dispatches the work. This decouples the protocol from business data, allowing other modules (e.g., config) to reuse the same framework by registering their own processors.

Two‑Stage Task Model (The Most Ingenious Part)

Stage 1 – Delay Merge (DelayTask)

When DistroProtocol.sync() is invoked, it does not send a network request immediately. It creates a DistroDelayTask and enqueues it into DistroDelayTaskExecuteEngine. If multiple changes for the same DistroKey arrive within a configurable window (default 1000 ms, property nacos.core.protocol.distro.data.sync.delayMs), the merge() method combines them, keeping only the latest action and creation time:

@Override
public void merge(AbstractDelayTask task) {
    DistroDelayTask oldTask = (DistroDelayTask) task;
    if (!action.equals(oldTask.getAction()) && createTime < oldTask.getCreateTime()) {
        action = oldTask.getAction();
        createTime = oldTask.getCreateTime();
    }
    setLastProcessTime(oldTask.getLastProcessTime());
}

This reduces three rapid client registrations (each would normally trigger a request) to a single network call.

Stage 2 – Execute Sync (ExecuteTask)

When the delay expires, DistroDelayTaskProcessor converts the delay task into a real execution task. For ADD or CHANGE actions it creates a DistroSyncChangeTask that:

Serializes the current client data from DataStorage.

Uses the TransportAgent (implemented as DistroClientTransportAgent with gRPC) to send the data to target nodes. The default gRPC timeout is 3000 ms.

Delete actions are handled similarly with a DistroSyncDeleteTask.

Incremental Sync – Event‑Driven + Responsibility Sharding

DistroClientDataProcessor

registers as a SmartSubscriber for three events:

@Override
public List<Class<? extends Event>> subscribeTypes() {
    result.add(ClientEvent.ClientChangedEvent.class);
    result.add(ClientEvent.ClientDisconnectEvent.class);
    result.add(ClientEvent.ClientVerifyFailedEvent.class);
    return result;
}

On a ClientChangedEvent it calls isInvalidClient() to ensure the client is temporary and that the current node is responsible for it. If both checks pass, it invokes distroProtocol.sync(). This “responsibility sharding” guarantees that each client has exactly one write source, preventing conflicting updates.

Periodic Verification – Version Reconciliation + Auto‑Repair

DistroVerifyTimedTask

runs every 5000 ms (default). It gathers clientId and revision for all clients the node is responsible for, sends a verify request to other nodes, and compares versions. If versions match, the check passes; otherwise a ClientVerifyFailedEvent is published.

@Override
public void run() {
    List<Member> targetServer = serverMemberManager.allMembersWithoutSelf();
    for (String each : distroComponentHolder.getDataStorageTypes()) {
        verifyForDataStorage(each, targetServer);
    }
}

When verification fails, the callback publishes ClientVerifyFailedEvent, which triggers a direct sync with zero delay:

private void syncToVerifyFailedServer(ClientEvent.ClientVerifyFailedEvent event) {
    distroProtocol.syncToTarget(distroKey, DataOperation.ADD, event.getTargetServer(), 0L);
}

New Node Join – Full‑Snapshot Loading

When a new Nacos node starts with an empty memory, it runs DistroLoadDataTask. The task first starts periodic verification and then initiates a full‑snapshot load:

private void startDistroTask() {
    if (EnvUtil.getStandaloneMode()) { isInitialized = true; return; }
    startVerifyTask();
    startLoadTask();
}

The load process contacts existing nodes, fetches a snapshot via transportAgent.getDatumSnapshot(), hands it to the processor, and marks the storage as initialized. If loading fails, it retries after loadDataRetryDelay (default 30 s).

Failure Retry – Fixed Delay, Not Exponential Backoff

The retry handler creates a new DistroDelayTask with a fixed retry delay (default 3000 ms) and re‑queues it:

@Override
public void retry(DistroKey distroKey, DataOperation action) {
    DistroDelayTask retryTask = new DistroDelayTask(distroKey, action,
        DistroConfig.getInstance().getSyncRetryDelayMillis());
    distroTaskEngineHolder.getDelayTaskExecuteEngine().addTask(distroKey, retryTask);
}

The design assumes failures are temporary node outages, so a quick fixed‑delay retry restores consistency faster than exponential backoff.

Full Timeline of a Registration

T=0   Client gRPC registers instance
      → EphemeralClientOperationServiceImpl.registerInstance()
      → Client.addServiceInstance()
      → NotifyCenter.publishEvent(ClientChangedEvent)
T=0   DistroClientDataProcessor.onEvent()
      → isResponsibleClient()? YES
      → distroProtocol.sync(clientId, CHANGE)
T=0   DistroDelayTask enqueued (delay=1000ms)
0‑1000ms  Same client changes merge into existing DelayTask
T=1000 DistroDelayTaskProcessor processes
      → DistroSyncChangeTask created
      → getDistroData(): serialize full client data
      → TransportAgent.syncData(): gRPC send to other nodes
T≈1xxx Remote node receives data → distroProtocol.onReceive()
      → DistroClientDataProcessor.processData()
      → clientManager.syncClientConnected() + upgradeClient()
Every 5s DistroVerifyTimedTask runs → version reconciliation → if mismatch, trigger full sync

Five Core Design Philosophies

Delay Merge : DelayTask (default 1000 ms) + merge() → high‑frequency changes do not flood the network.

Responsibility Sharding : isInvalidClient() check → each client has a single write entry.

Version Reconciliation : DistroVerifyTimedTask every 5 s + revision comparison → fallback repair of inconsistencies.

Full Snapshot : DistroLoadDataTask with retry delay 30 s → smooth onboarding of new nodes.

Protocol Decoupling : ComponentHolder + SPI pattern → same framework supports multiple business types.

Conclusion

Distro is not a generic gossip protocol, not Raft, and not a new consensus algorithm. It is a purpose‑built AP synchronization protocol for the service‑registration scenario, embodying the principle “each node only handles its own slice and then tells others.”