Master Hadoop High Availability: A Complete Step‑by‑Step HA HDFS & YARN Guide

Introduction

Big data remains a hot field, and its core value is data. This guide is based on the open‑source "Big Data Notes" repository (https://github.com/heibaiying/BigData-Notes) and focuses on Hadoop high availability.

1. High Availability Overview

Hadoop High Availability (HA) consists of HDFS HA and YARN HA. While both follow similar principles, HDFS NameNode requires stricter consistency guarantees, making its implementation more complex.

1.1 Overall Architecture

The HDFS HA architecture includes:

Active NameNode and Standby NameNode (mutual backup)

ZKFailoverController – controls failover, monitors health, and triggers elections via ZooKeeper

ZooKeeper ensemble – provides quorum for leader election

Shared storage system – stores synchronized metadata for both NameNodes

DataNode – reports block locations to both NameNodes

Image source: https://www.edureka.co/blog/how-to-set-up-hadoop-cluster-with-hdfs-high-availability/

1.2 QJM‑Based Shared Storage Synchronization

Hadoop can use Quorum Journal Manager (QJM) or NFS as the shared storage. With QJM, the Active NameNode writes EditLogs to a JournalNode cluster; the Standby NameNode periodically syncs them. Failover succeeds when a majority of JournalNodes have persisted the log, so an odd number of nodes (minimum 3) is required, and the system can tolerate up to N failures in a 2N+1 cluster.

1.3 NameNode Failover Process

HealthMonitor initializes and periodically calls HAServiceProtocol RPC to check NameNode health.

If health changes, HealthMonitor notifies ZKFailoverController.

ZKFailoverController triggers ActiveStandbyElector for automatic election.

ActiveStandbyElector interacts with ZooKeeper to complete the election.

After election, ActiveStandbyElector informs ZKFailoverController of the new Active or Standby role.

ZKFailoverController invokes HAServiceProtocol RPC to switch the NameNode state.

1.4 YARN High Availability

YARN ResourceManager HA follows a similar pattern, but because ResourceManager holds less metadata, its state is stored directly in ZooKeeper, which also handles the leader election.

2. Cluster Planning

To meet HA requirements, the cluster must have at least two NameNodes (active + standby), two ResourceManagers, and three JournalNodes. The following diagram shows a three‑node layout.

3. Prerequisites

All servers must have JDK installed.

A ZooKeeper ensemble must be set up.

Password‑less SSH login must be configured between all servers.

4. Cluster Configuration

4.1 Download and Extract Hadoop

# tar -zvxf hadoop-2.6.0-cdh5.15.2.tar.gz

4.2 Set Environment Variables

# vim /etc/profile

export HADOOP_HOME=/usr/app/hadoop-2.6.0-cdh5.15.2

export PATH=${HADOOP_HOME}/bin:$PATH

# source /etc/profile

4.3 Edit Configuration Files

hadoop-env.sh

# Specify JDK location

export JAVA_HOME=/usr/java/jdk1.8.0_201/

core-site.xml

<configuration>
  <property>
    <name>fs.defaultFS</name>
    <value>hdfs://hadoop001:8020</value>
  </property>
  <property>
    <name>hadoop.tmp.dir</name>
    <value>/home/hadoop/tmp</value>
  </property>
  <property>
    <name>ha.zookeeper.quorum</name>
    <value>hadoop001:2181,hadoop002:2181,hadoop003:2181</value>
  </property>
  <property>
    <name>ha.zookeeper.session-timeout.ms</name>
    <value>10000</value>
  </property>
</configuration>

hdfs-site.xml

<configuration>
  <property>
    <name>dfs.replication</name>
    <value>3</value>
  </property>
  <property>
    <name>dfs.namenode.name.dir</name>
    <value>/home/hadoop/namenode/data</value>
  </property>
  <property>
    <name>dfs.datanode.data.dir</name>
    <value>/home/hadoop/datanode/data</value>
  </property>
  <property>
    <name>dfs.nameservices</name>
    <value>mycluster</value>
  </property>
  <property>
    <name>dfs.ha.namenodes.mycluster</name>
    <value>nn1,nn2</value>
  </property>
  <property>
    <name>dfs.namenode.rpc-address.mycluster.nn1</name>
    <value>hadoop001:8020</value>
  </property>
  <property>
    <name>dfs.namenode.rpc-address.mycluster.nn2</name>
    <value>hadoop002:8020</value>
  </property>
  <property>
    <name>dfs.namenode.http-address.mycluster.nn1</name>
    <value>hadoop001:50070</value>
  </property>
  <property>
    <name>dfs.namenode.http-address.mycluster.nn2</name>
    <value>hadoop002:50070</value>
  </property>
  <property>
    <name>dfs.namenode.shared.edits.dir</name>
    <value>qjournal://hadoop001:8485;hadoop002:8485;hadoop003:8485/mycluster</value>
  </property>
  <property>
    <name>dfs.journalnode.edits.dir</name>
    <value>/home/hadoop/journalnode/data</value>
  </property>
  <property>
    <name>dfs.ha.fencing.methods</name>
    <value>sshfence</value>
  </property>
  <property>
    <name>dfs.ha.fencing.ssh.private-key-files</name>
    <value>/root/.ssh/id_rsa</value>
  </property>
  <property>
    <name>dfs.ha.fencing.ssh.connect-timeout</name>
    <value>30000</value>
  </property>
  <property>
    <name>dfs.client.failover.proxy.provider.mycluster</name>
    <value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
  </property>
  <property>
    <name>dfs.ha.automatic-failover.enabled</name>
    <value>true</value>
  </property>
</configuration>

yarn-site.xml

<configuration>
  <property>
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle</value>
  </property>
  <property>
    <name>yarn.log-aggregation-enable</name>
    <value>true</value>
  </property>
  <property>
    <name>yarn.log-aggregation.retain-seconds</name>
    <value>86400</value>
  </property>
  <property>
    <name>yarn.resourcemanager.ha.enabled</name>
    <value>true</value>
  </property>
  <property>
    <name>yarn.resourcemanager.cluster-id</name>
    <value>my-yarn-cluster</value>
  </property>
  <property>
    <name>yarn.resourcemanager.ha.rm-ids</name>
    <value>rm1,rm2</value>
  </property>
  <property>
    <name>yarn.resourcemanager.hostname.rm1</name>
    <value>hadoop002</value>
  </property>
  <property>
    <name>yarn.resourcemanager.hostname.rm2</name>
    <value>hadoop003</value>
  </property>
  <property>
    <name>yarn.resourcemanager.webapp.address.rm1</name>
    <value>hadoop002:8088</value>
  </property>
  <property>
    <name>yarn.resourcemanager.webapp.address.rm2</name>
    <value>hadoop003:8088</value>
  </property>
  <property>
    <name>yarn.resourcemanager.zk-address</name>
    <value>hadoop001:2181,hadoop002:2181,hadoop003:2181</value>
  </property>
  <property>
    <name>yarn.resourcemanager.recovery.enabled</name>
    <value>true</value>
  </property>
  <property>
    <name>yarn.resourcemanager.store.class</name>
    <value>org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore</value>
  </property>
</configuration>

mapred-site.xml

<configuration>
  <property>
    <name>mapreduce.framework.name</name>
    <value>yarn</value>
  </property>
</configuration>

slaves

hadoop001
hadoop002
hadoop003

5. Starting the Cluster

5.1 Start ZooKeeper

zkServer.sh start

5.2 Start JournalNode

hadoop-daemon.sh start journalnode

5.3 Initialize NameNode

hdfs namenode -format

After formatting, copy the metadata directory to the standby NameNode (e.g., from hadoop001 to hadoop002):

scp -r /home/hadoop/namenode/data hadoop002:/home/hadoop/namenode/

5.4 Initialize HA State in ZooKeeper

hdfs zkfc -formatZK

5.5 Start HDFS

start-dfs.sh

5.6 Start YARN

start-yarn.sh

Manually start the second ResourceManager if it is not running:

yarn-daemon.sh start resourcemanager

6. Verifying the Cluster

6.1 Check Processes

Run jps on each node; expected processes include NameNode, DataNode, JournalNode, ZKFailoverController, ResourceManager, NodeManager, and QuorumPeerMain.

6.2 Access Web UI

HDFS UI: http://<em>hostname</em>:50070; YARN UI: http://<em>hostname</em>:8088 Example screenshots show the active NameNode, standby NameNode, active ResourceManager, standby ResourceManager, and JournalNode status.

7. Restarting the Cluster

After the initial setup, subsequent restarts are simpler (assuming ZooKeeper is already running):

On the primary node, run start-dfs.sh to bring up all HDFS HA services.

On the secondary node, run start-yarn.sh to start YARN.

If the third ResourceManager is not active, start it manually with yarn-daemon.sh start resourcemanager.

References

HDFS High Availability Using the Quorum Journal Manager – https://hadoop.apache.org/docs/r3.1.2/hadoop-project-dist/hadoop-hdfs/HDFSHighAvailabilityWithQJM.html

ResourceManager High Availability – https://hadoop.apache.org/docs/r3.1.2/hadoop-yarn/hadoop-yarn-site/ResourceManagerHA.html

IBM Developer article “Hadoop NameNode High Availability (HA) Implementation Analysis” – https://www.ibm.com/developerworks/cn/opensource/os-cn-hadoop-name-node/index.html