Design and Implementation of a Handcrafted Distributed Cluster (MyCluster)

The goal of this section is to manually design and develop a challenging distributed cluster named MyCluster without relying on any distributed frameworks or middleware, in order to strengthen core architectural skills.

Requirements include building MyCluster from scratch in any language, using a master‑slave structure where the Master node collects user tasks (MyTask) and dispatches them, handling node failures, task retries, and state recovery, and supporting Master node election when the current Master fails.

MyCluster nodes share a common configuration file, with some parameters (e.g., ports, thread pool size) varying per machine; the configuration must be centrally managed and versioned.

Leader election is achieved by assigning each node a unique name sorted alphabetically; the node with the smallest name becomes the Leader. If the Leader fails, the next smallest alive node takes over, avoiding complex algorithms.

In case of a split‑brain scenario, the cluster either pauses or lets the larger partition (more than half of the nodes) continue serving, simplifying decision making when the node count is fixed.

Configuration management is split into a static seed file (node IDs, IPs, ports) and a dynamic shared configuration file that the Master updates and distributes, with version numbers to support rollback and reduce transmission frequency.

The update process follows a two‑phase commit: (1) Master pushes the new configuration to all slaves, (2) slaves validate and acknowledge, (3) Master sends a commit command; slaves that cannot apply the update stop to avoid inconsistency.

Communication between nodes uses long‑lived TCP connections with binary messages. Each message includes a type identifier, sequence number, and protocol version to handle asynchronous responses and future upgrades.

Message types include HelloMessage (node ID, uptime, cluster state, current Master ID, config version), LeaderDeclare (Master ID, online node count, config version), FollowKing (node ID, whether it needs a new config), WaitInPlace (node ID, online node count) with corresponding replies, and ConfigMessage (config version, compression flag, content).

State transitions are defined as Leader‑Select, Working, and Standby. The Master monitors node joins/leaves, triggering re‑election or state changes as needed, and uses periodic HelloMessage heartbeats that can also report resource usage (memory, CPU, disk, bandwidth).

Management commands include node offline, topology report, and configuration update. A client can be a traditional CLI or a lightweight web server exposing REST interfaces for external integration.

The complete architecture of MyCluster is illustrated in the final diagram.