Analysis of Apache Spark's Unified Memory Management Model (Spark 2.2.1)

This article examines the memory management model of Apache Spark, specifically the UnifiedMemoryManager introduced in Spark 2.2.1. It concentrates on the executor-side memory layout and does not include source‑code listings.

On‑heap Memory

By default Spark uses only on‑heap memory, which is divided into four main regions:

Execution Memory : stores temporary data for shuffle, join, sort, aggregation, etc.

Storage Memory : holds cached RDD data and unrolled data.

User Memory : keeps metadata such as RDD dependencies required by transformations.

Reserved Memory : a fixed portion reserved for internal Spark objects (300 MB in Spark 2.2.1).

The total on‑heap memory available to an executor is calculated as usableMemory = systemMemory - reservedMemory, where systemMemory corresponds to Runtime.getRuntime().maxMemory() (configured via spark.executor.memory or --executor-memory).

Off‑heap Memory

Since Spark 1.6, off‑heap memory can be enabled using spark.memory.offHeap.enabled and sized with spark.memory.offHeap.size. Off‑heap memory is allocated via Java’s unsafe API (similar to C’s malloc) and bypasses the JVM garbage collector, reducing GC overhead but requiring manual allocation and release.

When off‑heap memory is enabled, both on‑heap and off‑heap regions coexist, and the Execution and Storage memory pools are the sum of their respective on‑heap and off‑heap portions.

Dynamic Adjustment Between Execution and Storage Memory

Earlier Spark versions allocated Execution and Storage memory statically, which could lead to under‑utilisation. Modern Spark allows the two pools to share free space dynamically: if Execution memory is short but Storage has spare capacity, Execution can borrow from Storage, and vice‑versa (except that Storage cannot currently return borrowed space to Execution).

The implementation works as follows:

Initial Execution and Storage sizes are set via spark.memory.storageFraction.

If both pools run out of space, data is spilled to disk using an LRU policy.

When one pool needs space and the other has free memory, the needy pool can borrow that space; Execution can return borrowed memory after spilling it, while Storage currently cannot.

Task‑level Memory Distribution

Each executor maintains a HashMap<TaskId, Long> tracking the memory used by individual tasks. When a task requests numBytes from Execution memory, the map is consulted and updated. If sufficient free memory exists, the request succeeds; otherwise the task blocks until enough memory is released. The per‑task allocation range is roughly 1/(2N) ~ 1/N of the total Execution memory, where N is the number of concurrent tasks.

For example, with 10 GB Execution memory and 5 running tasks, each task may obtain between 1 GB and 2 GB.

Practical Example – Only On‑heap Memory

Configuration: --executor-memory 18g Without explicit spark.memory.fraction settings, the Spark UI shows about 10.1 GB of Storage memory. The calculation follows the formulae described earlier, as illustrated by the following screenshots:

Further details on the byte‑to‑GB conversion logic can be found in Spark’s /core/src/main/resources/org/apache/spark/ui/static/utils.js file:

Practical Example – Enabling Off‑heap Memory

When off‑heap memory is turned on, the configuration might look like:

The article concludes by encouraging readers to like, bookmark, and share the content.