HDFS HA Architecture: Components, ZKFC Failover, and Split-Brain Prevention

HDFS HA Architecture Overview

HDFS High Availability (HA) eliminates the single point of failure by deploying Active and Standby NameNodes. The architecture achieves strong metadata consistency through a Shared Edits Log and uses the ZKFC (ZooKeeper Failover Controller) for automatic health monitoring and failover.

Core Components and Roles

The HDFS HA architecture consists of the following core components working in concert:

(The article includes diagrams illustrating the component interactions.)

ZKFC Automatic Failover Process

The automatic failover is entirely driven by ZKFC, following these steps:

Continuous Monitoring : Each ZKFC continuously monitors the health of the NameNode on its node.

Failure Detection : When the Standby node's ZKFC detects that the Active NameNode has failed (e.g., process crash, network interruption), it triggers the failover process.

Lock Release and Preemption : The failed Active's ZKFC loses its ZooKeeper session and automatically deletes the ephemeral node (the "lock") it created. The Standby's ZKFC detects the lock release and attempts to create its own lock node in ZooKeeper.

Election and Fencing (Isolation) : The ZKFC that successfully creates the lock becomes the new Active. Before switching, it executes Fencing to forcibly isolate the old Active, ensuring it no longer writes data, thus preventing split-brain. Fencing methods include SSH to the old node to run kill -9 or executing custom scripts (e.g., sshfence, shellfence).

State Transition : After fencing completes, the ZKFC transitions the local Standby NameNode to Active state.

Service Recovery : The new Active NameNode loads all EditLogs, and DataNodes automatically register with it and report block information, restoring cluster service.

Split-Brain: Causes and Prevention

What Is Split-Brain?

Split-brain occurs in distributed systems when network failures partition the cluster, causing two or more nodes to simultaneously believe they are Active and serve requests . This leads to data corruption or loss .

Common Causes of Split-Brain

Network Partition : Switch failures, loose cables, etc., disrupt connectivity between Active/Standby NameNodes and the ZooKeeper cluster.

Heartbeat Timeout Misjudgment : Network latency causes heartbeat timeouts, leading Standby to incorrectly assume Active is down and initiate failover.

JVM or System Overload : Full GC or high system load prevents NameNode from responding promptly, causing false failure detection.

Fencing Mechanism Failure : Network or permission issues prevent successful isolation of the old Active.

ZooKeeper Issues : ZooKeeper cluster failures or election anomalies confuse ZKFC state.

Core Prevention Mechanisms and Best Practices

HDFS HA employs multiple layers to prevent split-brain; production environments must ensure these are correctly configured and operational.

Core Prevention Mechanisms

Shared Storage "Majority Acknowledgment" : The Active NameNode must receive acknowledgments from a majority of JournalNodes before committing edits. During a network partition, the old Active cannot obtain a majority and is forced to step down.

ZooKeeper Distributed Lock : ZKFC uses ZooKeeper ephemeral nodes to implement a distributed lock, guaranteeing global uniqueness. If a node loses connection to ZooKeeper, it automatically releases the lock and its NameNode is demoted.

Forced Fencing (Isolation) : As a last line of defense, before the new Active takes over, the system forcibly terminates the old Active process via SSH or other methods. Multiple fencing methods can be configured, such as sshfence or shellfence.

Production Best Practices

Configure Reliable Fencing : Properly set dfs.ha.fencing.methods.

Ensure Time Synchronization : Deploy NTP on all nodes to keep clocks consistent.

Set Reasonable Timeouts : Adjust parameters like dfs.ha.zkfc.port and ha.zookeeper.session-timeout.ms based on network conditions to balance failover speed and misjudgment risk.

Comprehensive Monitoring and Alerting : Focus on NameNode status, ZKFC logs, ZooKeeper cluster health, and network conditions.

Regular Failover Drills : Periodically conduct failover exercises to validate the HA process.