Hadoop NameNode Performance Bottlenecks and Solutions: Federation, ViewFS, FastCopy, Balance & Mover

The current Hadoop cluster runs version 2.7.3, while the latest production release is Hadoop 3.2.0. As the cluster grows, the single NameNode faces memory, performance, and restart‑time bottlenecks: the heap holds ~118 GB for 315 M namespaces and 305 M blocks, RPC QPS is limited by coarse‑grained metadata locks, and restarting the NameNode can take hours.

To address these issues, the team adopts a solution based on HDFS federation, ViewFS, and FastCopy, combined with optimized Balance and Mover operations.

Big Data Architecture

Federation

Federation expands the namespace horizontally by using multiple independent NameNodes, each managing its own namespace. All NameNodes share the same DataNodes, which register with every NameNode and periodically send heartbeats and block reports.

Each NameNode belongs to a cluster identified by a clusterId . Different clusterId s indicate separate clusters. A block pool is a logical grouping of blocks belonging to a single namespace; a DataNode can host blocks from multiple block pools.

ViewFS

ViewFS provides a unified view over multiple HDFS namespaces, making it useful in a federated environment with several NameNodes.

Reference: Hadoop ViewFS documentation

MountTable

The mount table aligns HDFS directories with ViewFS directories, ensuring consistent path mappings.

FastCopy

FastCopy accelerates data copying by creating hard links on the same DataNode, achieving orders of magnitude higher throughput than the standard distcp . The implementation modifies the copy logic to use hard links and adds support for ACLs, storage policies, and multithreaded listing.

hadoop distfastcp -strategy dynamic -prbugpcaxt -update -skipcrccheck -i -delete
hdfs://ip:9000/$1    hdfs://ip:9000/$1
FastCopy.CopyResult c = fcp.copy(sourceFileStatus.toString(), target.toString(),
    (DistributedFileSystem) srcFileSys, (DistributedFileSystem) targetFS);

FastCopy’s performance is compared against distcp in the following chart:

Balance & Mover

The balancer redistributes blocks when some DataNodes become full or new nodes join the cluster. The mover enforces storage policies by relocating blocks that do not match the target policy.

Because the cluster uses ZFS with transparent compression, mover operations can quickly fill disks. By limiting mover activity to specific DataNodes and running balance only after disk usage drops below 85 %, the team mitigates this issue.

for (DatanodeStorageReport r : reports) {
    final DDatanode dn = dispatcher.newDatanode(r.getDatanodeInfo());
    LOG.info("DFSConfigKeys.DFS_BALANCE_ONLY_DISK :" + typeStore);
    if (this.typeStore) {
        for (StorageType t : StorageType.getMovableTypes()) {
            if ("DISK".equals(t.name())) {
                // ...
            }
        }
    } else {
        for (StorageType t : StorageType.getMovableTypes()) {
            // ...
        }
    }
}

Testing, Compatibility, and Patches

Functional and compatibility tests verify the new federation, ViewFS, and FastCopy deployments. Relevant JIRA patches include HADOOP‑12253, HIVE‑11364, HIVE‑10790, HIVE‑11920, and HDFS‑2139 (FastCopy hard‑link updates).

Data Partitioning Strategy

Two main partitioning approaches are used: (1) partition by HDFS directory to split NameNode traffic, and (2) partition by table partitions to isolate historical cold data. The team chose to migrate the staging layer (stg) data to a new NameNode, handling ingestion tools (Flume, Sqoop, Databus) and moving YARN log‑aggregation directories as well.

Domain Compatibility and ViewFS Rollout Steps

Operational measures address ViewFS compatibility issues across Hadoop, Hive, Tez, Spark, Flume, and Sqoop clients. The rollout steps include identifying affected clients via HDFS audit logs, upgrading client configurations, adjusting Flume temporary directories, and performing staged fast‑copy migrations.

Summary & Outlook

Federation Phase 1 migrated the staging layer, reducing NameNode memory usage by ~30 %. Future work will extend migration to other business domains, explore Router‑Based Federation (RBF), and further isolate mount‑table changes from client impact.

RF can run multiple DfsRouter processes, retrieve mount information from ZooKeeper, and monitor NameNode health and HA status.

About Us

The Big Data Architecture Team at Beike Zhaofang is responsible for the company’s data storage, compute, and real‑time streaming platforms, delivering a stable, efficient, and open big‑data foundation.