Tackling HDFS Performance Bottlenecks: Real‑World Optimizations from VIP.com

1. Performance Challenges

HDFS can scale to thousands of nodes and >100 PB. After upgrading the Hadoop cluster from 2.5.0‑cdh5.3.2 to 2.6.0‑cdh5.13.1, the NameNode RPC queue time degraded weekly, sometimes taking minutes, causing Hive jobs to fail with errors such as “Unable to close file because the last block does not have enough number of replicas.” Restarting the cluster temporarily alleviated the issue, but a root‑cause fix was needed.

2. Performance Optimizations

The slowdown originates from the NameNode’s throughput limit: each write acquires an exclusive write lock, forcing other operations to wait in the RPC queue. Two optimization tracks were pursued: code‑level changes and business‑level configuration.

2.1 Delayed Datanode Block Reports

Datanodes normally report each newly written block to the NameNode immediately, causing a flood of lock‑acquiring requests. By configuring a delayed block‑report interval, multiple block reports are batched, reducing lock contention.

<property>
  <name>dfs.blockreport.incremental.intervalMsec</name>
  <value>300</value>
</property>

In our cluster the value is set to 300 ms, so a datanode waits 300 ms before sending the next report, allowing the NameNode to process other RPCs faster.

2.2 Configurable Block Deletion Batch Size

Large‑scale deletions also hold the NameNode lock for long periods. Introducing a configurable batch size lets other requests obtain the lock more frequently.

<property>
  <name>dfs.namenode.block.deletion.increment</name>
  <value>1000</value>
  <description>
    The number of block deletion increment.
    This setting will control the block increment deletion rate to
    ensure that other waiters on the lock can get in.
  </description>
</property>

The change has been contributed to the Hadoop community (JIRA HDFS‑13831).

3. HDFS Federation Migration

Operating multiple independent clusters leads to high O&M cost, unbalanced resource usage, and performance bottlenecks. Federation consolidates namespaces under a shared Cluster ID, allowing datanodes to register with multiple NameNodes.

Key issues encountered:

Inconsistent topology scripts caused registration errors (e.g., “You cannot have a rack and a non‑rack node at the same level”).

Incorrect du command on datanodes reported doubled capacity, breaking registration.

Both problems were resolved by synchronising the topology script and fixing the du command.

After migration, three NameNodes serve all datanodes, improving load distribution.

4. Client‑Side Monitoring & Temp‑Directory Sharding

4.1 HDFS Client Monitoring

Metrics such as rename and create RPC latency are collected from the client side to complement server‑side monitoring. Graphs show average rename latency and write latency, highlighting the impact of NameNode port 8022 and datanode performance.

4.2 Temp‑Directory Sharding

Heavy RPC traffic originates from directories such as /mr/staging, /tmp/Hive/HDFS/, and /bip/developer/vipdm. Balancing these directories across multiple clusters or moving them to a third cluster reduces contention.

Adjust Hive or scheduler to balance defaultFs across clusters.

Redirect heavy directories to a dedicated third cluster via Federation.

Implement dual reporting to automatically balance load.

4.3 Reducing Hive‑Generated RPCs

Hive’s insert/create operations generate temporary data that triggers many rename RPCs. Consolidating temporary data outside the table namespace and disabling unnecessary HDFS encryption checks further cut RPC volume.

5. Small‑File Management

Excessive small files degrade HDFS performance. Two complementary approaches are used:

HDFS Federation expands NameNode capacity horizontally, mitigating but not eliminating the small‑file issue.

File Merging – For Hive jobs, CombineHiveInputFormat groups small files into larger splits; mapred.min/max.split.size are tuned. ORC tables use the CONCATENATE command to merge at the stripe level.

Periodic jobs automatically merge historic small‑file outputs, and metrics show a noticeable reduction in file count.

6. Looking Ahead: Hadoop Ozone

Ozone, an object‑storage layer built on HDFS, aims to support billions of objects while retaining Hadoop’s reliability and consistency. It is being tracked as a potential long‑term solution for massive‑scale storage.