Mastering Spark Task Performance: A Deep Dive into JVM GC Optimization

1. Overview

During Spark task development, from code writing to deployment and maintenance, several optimization concerns arise: code efficiency, resource allocation, data skew, and GC issues. This article introduces Spark task GC strategy optimization, starting with JVM and GC fundamentals, and will later present a concrete Spark example.

2. JVM Basics

2.1 Main Components

JVM consists of class loader system, runtime data areas, execution engine, and native interface. The runtime data area includes method area, heap, Java stack, PC register, and native method stack.

When a class is loaded, its metadata goes to the method area; objects are allocated on the heap. Each thread has its own PC register and stack, which stores local variables, parameters, and intermediate results. Native methods use a separate native stack.

The stack is composed of frames; each frame corresponds to a method call and is popped after the method returns.

2.2 Memory Layout

JVM memory is divided into Young Generation, Old Generation, and Permanent Generation (Metaspace in JDK 8+).

Young Generation : New objects are allocated here, split into Eden, Survivor1, Survivor2 with a typical ratio of 8:1:1. When Eden fills, a Minor GC moves surviving objects to a Survivor space; if Survivor cannot hold them, they are promoted to Old Generation.

Survivor Spaces : One of the two Survivor spaces is used at a time; the other remains empty, providing roughly 90% of young generation memory for allocation.

Old Generation : Holds long‑lived objects after surviving several GC cycles. Default ratio of Young to Old is 1:2, adjustable via -XX:NewRatio.

Metaspace : Stores class metadata, replacing the Permanent Generation in JDK 8.

3. GC Fundamentals

3.1 Concepts

JVM manages heap and non‑heap memory. GC tracks object allocation and reachability, using a directed graph to identify live objects.

3.2 GC Process

(1) When Eden is full, a Minor (Young) GC copies surviving objects to a Survivor space.

(2) After objects survive long enough or Survivor2 fills, they are promoted to Old Generation.

(3) When Old space approaches capacity, a Full GC is triggered.

Full GC typically occurs when Old space is exhausted, System.gc() is called, or heap allocation policies change after a previous GC.

In Spark, GC aims to keep the Old Generation for long‑lived RDDs while the Young Generation handles short‑lived objects, avoiding Full GC pauses.

4. Common GC Algorithms

4.1 Mark‑Sweep

Two phases: mark live objects, then sweep away unreachable ones. Suitable for many live objects, especially in Old Generation, but can cause memory fragmentation and requires two full scans.

4.2 Copying

Divides memory into two equal halves; live objects are copied from one half to the other when full, eliminating fragmentation. Efficient for Young Generation where most objects die quickly.

4.3 Mark‑Compact

After marking, live objects are compacted to one end of the heap, then free space is reclaimed, avoiding fragmentation without needing a second memory region.

4.4 Generational Collection

Separates heap into Young and Old generations, applying different algorithms: copying for Young, mark‑sweep or mark‑compact for Old.

5. Common Garbage Collectors (JDK 8)

5.1 Young Generation Collectors

Serial : Single‑threaded, uses Copying algorithm; pauses all application threads.

ParNew : Multithreaded version of Serial.

Parallel Scavenge (PS) : Multithreaded, aims for high throughput, also uses Copying.

5.2 Old Generation Collectors

Serial Old : Uses Mark‑Compact.

Parallel Old : Multithreaded, also Mark‑Compact.

CMS : Concurrent Mark‑Sweep, targets low pause times.

5.3 Whole‑Heap Collectors

G1 : Modern collector for server‑side workloads, balances pause time and throughput; recommended by Spark.

Throughput is defined as CPU time spent running user code divided by total CPU time (user code + GC). Short pause times benefit interactive services, while high throughput suits batch processing.

References

Spark tuning guide, GC articles, and related blog posts.