Mastering Hadoop YARN: CPU & Memory Management Strategies for Large‑Scale Clusters

1. History and Origin of YARN

Since Hadoop 2.x, Apache Hadoop YARN (Yet Another Resource Negotiator) has become the standard resource manager for Hadoop clusters, replacing the older JobTracker/TaskTracker architecture.

2. Related Resource‑Management Systems

Google Borg (the inspiration for Kubernetes)

Hadoop YARN itself

Twitter Mesos (open‑source)

YARN schedules MapReduce, Spark and other Hadoop applications.

3. On‑YARN Applications

Open‑source projects that run on YARN include:

Docker on YARN – https://conferences.oreilly.com/strata/strata-ca-2017/public/schedule/detail/55936

Presto YARN integration – https://prestodb.io/presto-yarn/ and https://github.com/prestodb/presto-yarn

Hoya – HBase on YARN (Hortonworks blog) – https://hortonworks.com/blog/introducing-hoya-hbase-on-yarn/

In most production clusters the dominant workloads are MapReduce and Spark; other on‑YARN applications are outside the scope of this summary.

4. Multi‑Tenant Design, Queues and Node Labels

YARN was designed for multi‑tenant, parallel applications. Early Hadoop used a FIFO scheduler; YARN enables multiple applications to run concurrently and share resources.

Resources are allocated via queues, each of which can be associated with node labels that group machines by characteristics (e.g., CPU‑rich, memory‑rich, GPU‑enabled).

5. Common Real‑World YARN Issues

Resource allocation imbalance: Over‑provisioned queues cause some jobs to wait while other queues are under‑utilized.

Application waste: Jobs often request far more memory or CPU than they actually use (e.g., a job requesting 32 GB but averaging 514 MB).

Example command to submit a MapReduce job to a specific node label:

yarn jar /usr/iop/<code>iopversion</code>/hadoop-mapreduce/hadoop-mapreduce-examples.jar wordcount \
  -D mapreduce.map.node-label-expression="X" /tmp/source /tmp/result

Example Spark submission with node‑label:

./bin/spark-submit --class org.apache.spark.examples.SparkPi \
  --master yarn --deploy-mode cluster \
  --driver-memory 3g --executor-memory 2g \
  --conf spark.yarn.am.nodeLabelExpression=Y \
  --conf spark.yarn.executor.nodeLabelExpression=Y \
  jars/spark-examples.jar 10

Node‑label support was introduced in Hadoop 2.6, remained experimental until Hadoop 2.8, and contained many bugs in early releases.

6. Node‑Label Bugs and Limitations

Bug 1: UI does not display node‑label resources in older CDH 5.7.3 / Hadoop 2.6 clusters.

Bug 2: The list all labels command is missing in some versions.

Bug 3: UI cannot show label‑based resource usage per queue.

7. Data‑Driven YARN Management

To address the above issues the team built automation tools that:

Collect time‑series metrics to evaluate queue usage.

Provide a UI for bulk, safe label assignment.

Generate clear charts showing queue capacity, historical usage and over‑commitment.

8. Automated Diagnosis with dr‑elephant

The open‑source LinkedIn dr‑elephant tool (https://github.com/linkedin/dr-elephant) provides diagnostics for Hadoop jobs. The team extended it to rank applications by memory waste within a queue, enabling users to self‑service before contacting administrators.

9. Usage Pattern Analysis and Operational Recommendations

Cluster‑wide usage shows daily peaks around midnight and troughs from 06:00 to 15:00. Many jobs are scheduled by cron‑like systems (e.g., Airflow’s default daily run at 00:00), causing resource contention. Recommendations:

Stagger low‑priority jobs to off‑peak hours.

Schedule monthly or quarterly “maintenance days” to adjust queue resources in bulk.

Use the trend charts to identify long‑term under‑utilized queues and re‑allocate capacity to overloaded queues.

Conclusion

Make resource‑allocation decisions data‑driven.

Develop flexible automation tools to reduce manual operations.

Prefer stable open‑source versions; treat experimental features (e.g., early node‑label support) as requiring patches and thorough testing.