Understanding Spark Checkpoint: Purpose, Mechanism, and Best Practices

Introduction – In production Spark jobs, the number of RDD transformations can be huge (e.g., tens of thousands) or individual transformations may be extremely time‑consuming, making it necessary to persist intermediate results.

Why Checkpoint? – Spark excels at multi‑step iterative algorithms and job reuse, but without reliable persistence repeated computation can waste resources. In‑memory persistence (persist) is fast but not fault‑tolerant; disk persistence can still suffer from hardware failures. Checkpoint provides a more reliable way to store data, typically on HDFS, leveraging its high‑availability features.

Checkpoint Goal – To guarantee absolute reliability for reusable RDDs, Spark checkpoints data to a location (usually HDFS) with multiple replicas, ensuring fault tolerance and high availability.

When to Use Checkpoint – It is applied to specific points in an RDD lineage where data must be persisted before further processing, breaking the lineage and enabling fault‑tolerant recovery.

Checkpoint Execution Diagram – (Illustration omitted) shows the flow from RDD computation to HDFS storage.

Source Code Analysis

1. RDD.iterator first checks the cache for checkpointed data, then falls back to the checkpoint storage.

2. SparkContext.setCheckpointDir defines the HDFS directory where checkpoint data will be written; multiple directories can be specified for efficiency.

3. When an RDD is checkpointed, all its parent RDDs are cleared. It is recommended to cache the RDD in memory (or local disk) before checkpointing; alternative storage such as Tachyon can also be used.

4. Before invoking checkpoint(), a persist() call is advised because checkpointing is lazy and only triggers after a job execution completes.

5. Checkpoint changes the RDD lineage, breaking the original dependency chain.

6. Invoking checkpoint() creates an RDDCheckpointData object.

7. After a job runs on the RDD, the checkpoint data’s checkpoint() method is called, which internally executes doCheckpoint() (or ReliableRDDCheckpointData.doCheckpoint() in production).

8. In production, ReliableRDDCheckpointData.writeRDDToCheckpointDirectory runs a job to write the RDD data to the checkpoint directory, producing a ReliableCheckpointRDD instance.

Best Practices – Always persist the RDD before checkpointing, understand that checkpointing is lazy and requires a downstream job to materialize, and be aware that checkpointing modifies lineage and creates additional metadata objects.

Source: https://www.cnblogs.com/itboys/p/9198504.html (author: 大葱拌豆腐)