Hadoop System Bottleneck Detection and MapReduce Optimization Guide

Detect System Bottlenecks

Performance tuning

Create a baseline to evaluate the cluster's initial performance with default configuration.

Analyze Hadoop counters, modify configurations, and re‑run jobs to compare against the baseline.

Repeat step 2 until the highest efficiency is achieved.

Identify Resource Bottlenecks

Memory bottleneck: frequent virtual memory swapping indicates insufficient memory.

CPU bottleneck: processor load >90% (or >50% on multi‑processor systems) and possible single‑thread CPU hog.

IO bottleneck: disk activity >85% (may be caused by CPU or memory issues).

Network bandwidth bottleneck: occurs during map‑to‑reduce shuffle when pulling data.

Identify Weak Resource Points

Check Hadoop cluster node health

Inspect JobTracker for black‑list, gray‑list, and excluded nodes.

Gray‑list nodes intermittently fail and should be repaired or excluded.

Check input data size

Larger input increases job runtime.

Examine counters such as HDFS_BYTES_WRITTEN , Reduce shuffle bytes , Map output bytes , Map input bytes .

Check massive IO and network blocking

Network or IO bottlenecks cause compute resources to wait.

Inspect FILE_BYTES_READ and HDFS_BYTES_READ to determine input‑related issues.

Inspect Bytes Written and HDFS_BYTES_WRITTEN to determine output‑related issues.

Compress data and use a combiner to reduce traffic.

Check for insufficient concurrent tasks

Idle CPU cores indicate under‑utilization.

Low network utilization also points to insufficient parallelism.

Check CPU oversaturation

Low‑priority tasks waiting for high‑priority ones cause excessive context switches.

Use vmstat to view context‑switch count (cs).

Oversaturation may stem from too many tasks on a host.

Strengthen Map & Reduce Tasks

Strengthen Map tasks

Determine write file size and processing time per map.

Large spill records cause performance issues; compare Map output records < Spilled Records.

Allocate memory buffers precisely.

Binary and compressed files are not splittable; treat them as whole.

Many small files generate excessive map tasks and waste resources.

Best practice: pack small files into larger containers (e.g., Avro, HAR, SequenceFile).

Large input files require larger block sizes; too small blocks increase mapper count.

Large blocks speed up disk IO but increase network overhead, potentially causing spill during map.

Map task workflow: read, map, spill, fetch, merge.

Read phase: read fixed‑size (64 MB) blocks from HDFS.

Map phase: measure map function execution time and record count; detect abnormal data or too many/few files.

Spill phase: locally sort data, partition by reducer, apply combiner if available, write to disk.

Fetch phase: buffer map output in memory and record intermediate data size.

Merge phase: each reducer merges map outputs into a single spill file.

Strengthen Reduce tasks

Compress, sort, and merge data (combiner, compression, filtering).

Address local disk and network issues.

Maximize memory allocation to keep data in RAM rather than spilling.

Slow Reduce may be caused by unoptimized reduce function, hardware problems, or bad Hadoop settings.

Calculate throughput by dividing shuffle input size by Reduce runtime.

Reduce workflow: shuffle, reduce, write.

Measure Reduce throughput and improve execution phase.

Shuffle phase: Map tasks transfer intermediate data to reducers, merging and sorting it.

Reduce phase: run reduce function on each key and its values, measuring time.

Write phase: output results to HDFS.

Optimize MapReduce Parameters

Use Combiner

Acts like a local Reduce to improve global Reduce efficiency.

Reduce function can serve as Combiner if it satisfies commutative and associative properties.

Combiner aggregates map output until its buffer fills, then sends data to reducers, greatly improving performance on large datasets.

Use Compression

Input compression: beneficial when large data is repeatedly processed; Hadoop auto‑detects suitable file extensions.

Compress Mapper output: reduces shuffle traffic and network load.

Compress Reducer output: lowers storage size and downstream input volume.

Enabling compression at any stage (input, map, or reduce) mitigates IO and network bottlenecks.

Use Correct Writable Types

FileInputFormat for raw bytes outperforms WritableComparable.

Prefer Text over String to avoid costly string splitting.

VIntWritable/VLongWritable can be faster than primitive int/long.

Choosing appropriate Writable types improves overall MR job performance.

Key comparison during Shuffle/Sort can become a bottleneck.

Reuse Objects

Reusing existing instances is cheaper than creating new ones.

Avoid short‑lived objects to reduce GC pressure.

Enable JVM reuse to lower overhead of launching new JVMs.

Optimize Mapper and Reducer Code

Achieve the same output with less time.

Achieve the same output with fewer resources.

Produce more output with the same resources in the same time.