How YARN and Kubernetes Solve Distributed Resource Management Challenges

Apache YARN

YARN (Yet Another Resource Negotiator) is a scalable cluster scheduling framework that provides high reliability. Its architecture includes a ResourceManager that controls the whole cluster and allocates resources to applications, and NodeManagers on each node that manage containers. The Application Master, acting as user code, requests resources from the ResourceManager and monitors container execution via NodeManagers.

Containers are abstractions of resources (CPU, memory, etc.) on a node, launched by the Application Master through the NodeManager.

A typical MapReduce job scheduling flow—submission, Application Master launch, resource request, and container‑based data processing—also applies to Spark, Flink and other engines, enabling unified management of short‑term compute tasks.

YARN offers three built‑in schedulers: FIFO, Fair, and Capacity. FIFO processes jobs in submission order, which is simple but unsuitable for production where jobs have different priorities.

Capacity Scheduler isolates resources into multiple queues for different organizations or users, allowing fair sharing and optional preemption to improve cluster throughput.

Fair Scheduler groups resources into pools with minimum guarantees and maximum limits, supports priority‑based allocation, and can preempt resources from over‑allocated pools to satisfy starving pools. It also provides load‑balancing based on task count.

Google Kubernetes

Kubernetes, an open‑source project from Google, manages Docker clusters and inherits Borg’s strengths. It abstracts CPU, GPU, memory, network, and storage across the cluster into a resource pool, enabling flexible scheduling based on real‑time resource availability.

The platform provides a unified scheduling framework capable of handling thousands of nodes and tens of thousands of containers, with extensible plug‑in interfaces for custom schedulers. Dynamic configuration is supported via ConfigMap, allowing applications to adapt at runtime.

Conclusion

This article introduced two distributed resource‑management technologies, YARN and Kubernetes. While YARN has proven effective for Hadoop clusters, its architecture limits evolution toward a universal data‑center scheduler. Since 2018, many projects (Spark, Flink, TensorFlow) have migrated from YARN to Kubernetes. The author’s company transitioned its internal big‑data platform from YARN to Kubernetes in 2017, and future articles will explore storage, compute, and scheduling in their data‑tech stack.