DSP’s Shift from Single‑Node to Distributed: A Deep Look at Its gRPC Cluster Architecture

1. Cluster Topology: Three Roles

DSP defines three cluster roles:

ConfigurationCenter (registration center) runs on port 3100, receives registration info from storage nodes and maintains the node list. Implemented by the dsp-register module.

StorageServer (storage node) runs on port 4100, registers itself to the registration center at startup and provides data storage. Implemented by the dsp-raft module.

Client (SQL client) connects to the FrontEnd on port 3016 and does not participate directly in cluster communication.

2. gRPC Communication Skeleton

GrpcServer: Server Abstraction

The GrpcServer class is the base for all gRPC servers. Its core method startRpc() builds and starts a server on the configured port:

public void startRpc() {
    ServerBuilder<?> builder = ServerBuilder.forPort(port);
    for (BindableService service : getService()) {
        builder.addService(service);
    }
    server = builder.build().start();
}

Sub‑classes only need to implement getService() to return the list of services to register. The server is kept alive by calling Thread.sleep(Long.MAX_VALUE) in block(), and it can be gracefully shut down via server.shutdownNow().

GrpcClient: Client Abstraction

GrpcClient

wraps the creation and management of a ManagedChannel. Its start() method supports two modes:

Plaintext mode: ManagedChannelBuilder.forAddress(host, port).usePlaintext() SSL mode: NettyChannelBuilder + InsecureTrustManagerFactory (currently a development‑only configuration without real certificates).

The close() method shuts the channel down with a 5‑second timeout.

3. Proto Definitions: Two RPC Services

RegisterAndHeartBeatService

service RegisterAndHeartBeatService {
    rpc registerNodeInfo(NodeRegisterRequest) returns(NodeRegisterReponse);
}

message NodeRegisterRequest {
    string hostname = 1;
    int32 port = 2;
}

message NodeRegisterReponse {
    int32 code = 1;
}

This is the only RPC exposed by the registration center. Storage nodes call registerNodeInfo with their hostname and port; a response code of 0 indicates success. A comment in the proto file ( //todo, we can register table/partition message) hints at future extensions for table‑level registration.

DdlService

service DdlService {
    rpc createTable(CreateTableRequest) returns(CreateTableResponse);
}

message CreateTableRequest {
    string table_name = 1;
}

The DDL service is intended for the registration center to push DDL operations to storage nodes. Currently the implementation returns UNIMPLEMENTED, serving as a placeholder.

4. Registration Flow: From Startup to Discovery

Storage Node Side

The StorageServer.start() method executes the following steps: startRpc(): starts the gRPC server (no services registered yet). initExecutionClient(): creates a RegisterAndHeartBeatRpcClient to talk to the registration center. registerToExecutionServer(): invokes rpcClient.registerLocation("127.0.0.1", port) to register itself. block(): blocks the main thread.

The client builds a NodeRegisterRequest and synchronously calls the registration RPC via a BlockingStub. Success (code = 0) is logged; failures are also logged.

Registration Center Side

When RegisterAndHeartBeatService.registerNodeInfo() receives a request, it:

Constructs a NodeRegisterReponse with code = 0.

Extracts hostname and port from the request.

Creates a HostAndPort value object and adds it to registerAndHeatBeatHandler.addNodeInfo().

Returns the response. HostAndPort implements equals() and hashCode() based on IP + port, ensuring correct de‑duplication in the internal Set<HostAndPort>.

5. Design Intent: Reserved Raft Consensus

The module name dsp-raft suggests an intention to implement the Raft consensus protocol, but the current code contains no leader election, log replication, or state machine logic. The structure, however, reserves space for these features.

Command Abstraction

public interface Command {
    CommandDetail getCommand();
}

public class CommandDetail {
    private final String key;
    private final String value;
    private final String table;
}

The Command interface and CommandDetail class represent Raft log entries, carrying key, value, and table. The only concrete implementation, UpdateCommand, returns null for getCommand(), indicating a placeholder for future write operations.

StorageBackend State Machine

public interface StorageBackend {
    default void store() { return; }
    default void query() { return; }
}

After a Raft log entry is committed, the state machine’s store() method applies the change. Two implementations are mentioned: RocksDbStorageBackend (marked with a //todo comment) and LucenceStorageBackend (empty), indicating future support for RocksDB or Lucene as the underlying storage engine.

DDL Distribution Path

The registration center defines RPC_CLIENT_MAP, a Map<HostAndPort, DdlServiceClient>, intended for broadcasting DDL commands to all storage nodes. Although the map is currently empty, the design states that the registration center acts as a DDL coordinator, using gRPC to ensure consistency across nodes.

6. Missing Pieces: From Skeleton to Usable System

Heartbeat Mechanism

Although the module name contains “HeartBeat”, there is no implementation. Comments such as //maybe should be register periodically and //heart beat// timeout//etc indicate that periodic registration/heartbeat is planned but not yet realized. A functional heartbeat would periodically send packets, let the registration center detect timeouts, and mark nodes offline.

Node Discovery

The registration center collects node information but does not expose a query interface. Storage nodes cannot discover peers, and clients cannot obtain a list of storage nodes for direct connections. Adding a listNodes RPC or a watch mechanism is required for node‑change notifications.

Raft Consensus

The biggest gap is the missing Raft implementation. Required components include:

Leader election (term increment, random election timeout, RequestVote RPC, follower/candidate/leader states).

Log replication ( AppendEntries RPC, log entry structure, matchIndex / nextIndex tracking, majority commit).

Snapshotting (log compaction, InstallSnapshot RPC).

Membership changes (joint consensus or single‑server changes).

The proto file therefore needs to add RequestVote and AppendEntries RPC definitions, and the StorageBackend interface must accept a Command parameter for applying log entries.

Data Routing

Currently the SlothTableEngine uses a random shard selection ( random.nextInt(shardNum)) and does not consider cluster topology. To achieve true distributed storage, the system must implement:

Shard routing based on a partition key to select the target node.

Distributed query execution that dispatches sub‑queries to multiple nodes and merges results.

Failover handling where a replica takes over when a node crashes.

7. Summary

The DSP cluster module provides a solid architectural skeleton:

Reusable gRPC communication layer ( GrpcServer / GrpcClient) – adding new RPCs only requires defining a proto.

Registration workflow is functional – storage nodes can register with the center, albeit as a one‑time operation.

Clear extension points – Command defines log format, StorageBackend defines the state‑machine interface, and DdlServiceClient defines the DDL distribution path.

From a middleware learning perspective, the article answers the key question: “What should the communication skeleton of a distributed database look like?” The answer is gRPC for transport, proto for interface definition, a registration center for service discovery, Command for log abstraction, and StorageBackend for the state machine. Populating this skeleton with a Raft implementation, heartbeat handling, and data routing would yield a complete distributed storage system.