Essential MapReduce, HBase, and Spark Configuration Parameters for Faster, More Stable Jobs

Background

MapReduce and Spark expose hundreds of configuration parameters; users can tune them for faster, more stable applications. This article lists the most commonly used parameters.

Key MapReduce Configuration Parameters

1. Resource‑related parameters

mapreduce.map.memory.mb

: Maximum memory (MB) a Map task can use, default 1024. Exceeding this kills the task. mapreduce.reduce.memory.mb: Maximum memory (MB) a Reduce task can use, default 1024. mapreduce.map.java.opts: JVM options for Map tasks, e.g., "-Xmx1024m -verbose:gc -Xloggc:/tmp/@[email protected]". mapreduce.reduce.java.opts: JVM options for Reduce tasks, similar format. mapreduce.map.cpu.vcores: Number of CPU cores for each Map task, default 1. mapreduce.reduce.cpu.vcores: Number of CPU cores for each Reduce task, default 1.

2. Fault‑tolerance parameters

mapreduce.map.maxattempts

: Maximum retries for a Map task, default 4. mapreduce.reduce.maxattempts: Maximum retries for a Reduce task, default 4. mapreduce.map.failures.maxpercent: Job fails if the percentage of failed Map tasks exceeds this value; default 0. Set >0 (e.g., 5) to tolerate some failures. mapreduce.reduce.failures.maxpercent: Same for Reduce tasks, default 0. mapreduce.task.timeout: Task timeout in milliseconds, default 300000. Increase if tasks run long.

3. Local‑mode parameters

Set the following for running MapReduce locally:

mapreduce.framework.name=local

mapreduce.jobtracker.address=local

fs.defaultFS=local

4. Efficiency and stability parameters

mapreduce.map.speculative

: Enable speculative execution for Map tasks, default false. mapreduce.reduce.speculative: Enable speculative execution for Reduce tasks, default false. mapreduce.job.user.classpath.first & mapreduce.task.classpath.user.precedence: Choose user‑provided JAR over Hadoop JAR when classes conflict, default false. mapreduce.input.fileinputformat.split.minsize: Minimum split size for file‑based input formats, default 134217728 (one block).

HBase Related Configuration Parameters

hbase.rpc.timeout

: RPC timeout, default 60s; avoid setting too low. ipc.socket.timeout: Socket connection timeout, should be ≤ RPC timeout, default 20s. hbase.client.retries.number: Number of retries, default 14, can be set to 3. hbase.client.pause: Pause between retries, default 1s, can be reduced (e.g., 100ms). hbase.regionserver.lease.period: Lease period for scan queries, default 60s.

Spark Related Configuration Parameters

1. Efficiency and stability

Enable intermediate result consolidation and speculation:

spark.shuffle.consolidateFiles=true

spark.speculation=true

2. Fault‑tolerance

Increase these values for better resilience:

spark.task.maxFailures=8

spark.akka.timeout=300

spark.network.timeout=300

spark.yarn.max.executor.failures=100