How to Master Hadoop Performance: A Real-World TPCx-HS Tuning Case Study

Hadoop Performance Tuning Case Study

Hadoop is a massive system whose tuning process is complex. Although Hadoop provides many tuning parameters, their sheer number makes it difficult for developers to choose the right ones, highlighting the challenge of optimization.

Test Case Introduction

TPCx-HS is a Hadoop performance benchmark provided by the TPC organization, used to evaluate hardware, software, and Hadoop filesystem compatibility in terms of performance, cost‑effectiveness, availability, and power consumption. It is essentially the TeraSort benchmark, sorting terabytes of data to test HDFS and MapReduce processing capabilities.

TeraGen : Generates large amounts of data and stores it in HDFS (Map only).

TeraSort : Reads the data from HDFS, sorts it using MapReduce, and writes the result back to HDFS (both Map and Reduce).

Validate : Verifies the sorted results; errors are reported if any file is not correctly ordered (both Map and Reduce).

Baseline Test Results

During TeraGen, the Map stage shows low CPU load while the network becomes the performance bottleneck.

During TeraSort, both the Map stage CPU and the Reduce stage network are fully saturated.

Validate shows stable performance with no obvious tuning needs.

Thus, the main tuning targets are TeraGen and TeraSort.

TeraGen Tuning Approach

The bottleneck in TeraGen is the network throughput caused by writing three HDFS replicas. The following principles guide the optimization:

Increase the slope of the left line: tasks should start quickly and finish promptly.

Keep the upper platform flat: the network must remain stable and fully utilized.

Avoid trailing on the right line: task workload should be balanced, and no single task should process excessive data.

To achieve these goals, several parameters are adjusted:

Increase the heartbeat frequency between service nodes to accelerate task startup.

Choose an appropriate block size based on the number of files, the number of Map tasks in TeraSort, and the execution time of each TeraGen task.

TeraSort Tuning Approach

TeraSort consists of Map and Reduce stages. The Map stage has high CPU load but low network load, while the Reduce stage has low CPU load but high network load, leading to idle waiting when stages run sequentially.

The optimization ideas are:

Compress and broadcast intermediate shuffle data to reduce I/O between Map and Reduce.

Balance system resources, the number of Map tasks, and Reduce tasks to increase overlap between stages.

Even with overlap, Map tasks may still dominate resources; therefore, the number of Reduce tasks should not be excessive.

Map/Reduce Task Numbers

The number of Map and Reduce tasks significantly impacts performance for both TeraGen and TeraSort. For a 10 TB example, the relationship among file count, split count, block size, and MapTask count is explained. NUM_MAPS determines the actual number of Map tasks in TeraGen, while TeraSort’s Map tasks are driven by total split count, and Reduce tasks are set by NUM_REDUCERS.

Hardware Environment Configuration

CPU : Set to Performance mode, ensure the CPU frequency stays at ≥2 GHz even when idle (check /proc/cpuinfo).

Network : Must operate at full speed; use network tools to verify throughput.

Disk : Sufficient disks are required; inadequate disk I/O becomes the bottleneck.

Conclusion

Beyond the parameter adjustments described above, memory and GC settings were also tuned during testing. As demonstrated, Hadoop tuning demands a solid understanding of the underlying architecture and the ability to align resource configurations with specific workload characteristics to achieve optimal performance.