Mastering Spark: Core Concepts, Architecture, Streaming & Performance Tuning

Spark Ecosystem and Execution Principles

Spark is widely used for large‑scale data processing such as advertising analytics, reporting, and recommendation systems because of its high performance, ease of use, and support for diverse workloads. It executes programs on a directed‑acyclic‑graph (DAG) engine that schedules transformations lazily and materializes results only when an action is invoked.

Spark Features

Speed: The DAG engine can be >10× faster than Hadoop MapReduce when reading from disk and >100× faster when data resides in memory.

Versatility: Supports batch analytics, real‑time streaming, graph processing, and machine learning.

Ease of Use: Provides >80 high‑level operators, multi‑language APIs (Scala, Java, Python, R), and connectors to many data sources.

Fault Tolerance: RDD lineage enables automatic recomputation; checkpointing adds durability.

Typical Workload Categories

Complex batch processing (minutes to hours).

Interactive queries on historical data (seconds to minutes).

Streaming data processing (hundreds of milliseconds to seconds).

Spark Architecture

The runtime follows a Master‑Slave model:

Driver: The entry point of an application; creates a SparkContext and coordinates execution.

Cluster Manager: Allocates resources (Standalone, Apache Mesos, Hadoop YARN).

Worker Nodes: Host one or more Executors that run tasks.

Executor: A JVM process that executes tasks on data partitions and stores intermediate results.

During execution the Driver serializes tasks, dependent JARs, and files, then ships them to Workers where Executors process the assigned partitions.

Cluster Deployment Modes

Standalone (independent Spark cluster).

Apache Mesos.

Hadoop YARN.

Core Concepts

Application: A Spark program consisting of a Driver and one or more Executors.

SparkContext: Schedules resources and creates RDDs.

Job: Triggered by an Action; may consist of multiple Stages.

Stage: A set of tasks that can run in parallel; boundaries are defined by shuffle operations.

Task: The smallest unit of work executed by an Executor.

RDD (Resilient Distributed Dataset): Immutable, partitioned collection that supports lineage‑based recovery.

DAGScheduler: Builds a DAG of Stages from Jobs.

TaskScheduler: Dispatches TaskSets to Executors.

Transformations: Lazy operations that return a new RDD.

Actions: Operations that trigger computation and return results to the Driver or write external storage.

RDD Details

RDDs are immutable, partitioned, and can be cached in memory for reuse. Fault recovery is achieved by recomputing lost partitions from their lineage.

Transformations and Actions

Transformations (e.g., map, filter, reduceByKey) build a new RDD without executing any computation. Actions (e.g., collect, count, saveAsTextFile) materialize the DAG, causing the lazy graph to be evaluated.

Jobs, Stages and Shuffle

Each Action launches a Job. Spark splits the Job into Stages at shuffle boundaries. During a shuffle (e.g., reduceByKey) intermediate data is written to disk as FileSegment s. By default each ShuffleMapTask creates R × M files (R reducers, M map tasks), which can cause high I/O and memory pressure.

Enabling file consolidation reduces the number of files: spark.shuffle.consolidateFiles=true With consolidation each core writes a single ShuffleFile, dramatically lowering file count and buffer memory usage.

Cache and Persistence

Cache marks an RDD for in‑memory storage; the actual caching occurs after the first Action that materializes the RDD. unpersist removes the cached data immediately.

val rdd1 = sc.textFile("hdfs://path/to/data")
rdd1.cache()
val rdd2 = rdd1.map(_.split(","))
val rdd3 = rdd1.filter(_ != "")
rdd2.take(10).foreach(println)
rdd3.take(10).foreach(println)
rdd1.unpersist()

Cache internally uses persist(StorageLevel.MEMORY_ONLY). Spark also supports other storage levels such as MEMORY_AND_DISK, DISK_ONLY, and serialized variants.

Spark Streaming Architecture

Spark Streaming converts an unbounded input stream into a series of micro‑batches processed by the same Spark engine. Four components are required:

A static RDD DAG template that encodes the processing logic.

A dynamic controller that slices incoming data into batches and instantiates the DAG for each batch.

A Receiver that ingests raw data and stores it in memory or on disk.

Fault‑tolerance mechanisms (checkpointing, write‑ahead logs) to recover from data loss or task failures.

Reference implementation: https://github.com/lw-lin/CoolplaySpark/blob/master/Spark%20Streaming%20%E6%BA%90%E7%A0%81%E8%A7%A3%E6%9E%90%E7%B3%BB%E5%88%97/0.1%20Spark%20Streaming%20%E5%AE%9E%E7%8E%B0%E6%80%9D%E8%B7%AF%E4%B8%8E%E6%A8%A1%E5%9D%97%E6%A6%82%E8%BF%B0.md#24

Best practices for streaming:

Avoid persisting intermediate data to disk on Workers; keep data in memory to maximize throughput.

Ensure each batch can be processed within its interval to prevent backlog.

Pre‑process incoming records (e.g., filtering) before they reach the task stage.

Resource Tuning

Each Executor’s memory is divided into three regions (default percentages):

Task Execution Memory: 20% – used by user code.

Shuffle Memory: 20% – buffers data exchanged between stages.

Storage Memory: 60% – holds cached RDD partitions and broadcast variables.

Adjust these fractions via spark.memory.fraction and spark.memory.storageFraction to match workload characteristics. Also tune the number of partitions per RDD so that each task processes a reasonable data slice, avoiding out‑of‑memory errors.

Additional tuning parameters include: spark.executor.cores – number of CPU cores per Executor. spark.default.parallelism – default number of partitions for RDDs without explicit partitioning. spark.shuffle.spill – controls whether shuffle data is spilled to disk.