Understanding Hadoop 2.0 High Availability: NFS vs QJM Explained

Overview

Before Hadoop 2.0, the NameNode was a single point of failure. Hadoop 2.0 introduces High Availability (HA) with two NameNodes: an active and a standby.

Basic Principle

The active NameNode writes edits to a local edits file and replicates them to either an NFS share or a set of JournalNodes (QJM). The standby periodically reads these edits, merges them with the fsimage to create a new fsimage, and notifies the active to adopt it. This keeps both NameNodes synchronized, allowing the standby to take over instantly if the active fails. Traditional secondary NameNode, checkpoint node, and backup node become unnecessary.

NFS Method

In the NFS approach, the active NameNode writes edits to a shared NFS directory, and the standby reads from it. The drawback is that network issues between a NameNode and the NFS server can break synchronization.

QJM (Quorum Journal Manager) Method

QJM replaces NFS with a quorum of JournalNodes (an odd number, e.g., 3,5,7…). The active NameNode writes edits to all JournalNodes; if a majority (n+1) acknowledge, the write is considered successful. The standby reads from the JournalNodes, providing fault tolerance that can survive up to n JournalNode failures.

Failover

Both manual and automatic failover are supported. Manual failover uses HA management commands to switch states. Automatic failover relies on ZooKeeper, which monitors the active NameNode’s health and promotes the standby when a failure is detected.

Practical Tips

QJM offers built‑in fencing and higher robustness; it is recommended for production.

JournalNodes consume little resources and can run on existing Hadoop cluster machines.

Further Reading

Ubuntu13.04 Hadoop setup

Ubuntu 12.10 + Hadoop 1.2.1 cluster configuration

Single‑node Hadoop installation on Ubuntu

Hadoop environment configuration on Ubuntu

Step‑by‑step Hadoop tutorial

Hadoop on Windows using virtual machines

References: http://hadoop.apache.org/docs/r2.2.0/hadoop-yarn/hadoop-yarn-site/HDFSHighAvailabilityWithNFS.html http://hadoop.apache.org/docs/r2.2.0/hadoop-yarn/hadoop-yarn-site/HDFSHighAvailabilityWithQJM.html