Why Spark Beats MapReduce: The RDD Story and Spark SQL Evolution

Guest Introduction

Meng Shuo is an expert in Hadoop and Spark, with experience at Oracle, Cloudera, and RedFlag‑Linux, now leading big‑data product development at Yunhe Enmo.

Live Session Overview

The session titled “Spark 2.0+ Technical Roadmap” introduced Spark’s background, motivations, and key components.

Origin of Spark and the RDD Paper

The speaker referenced the original Spark paper, whose title mentions “Resilient Distributed Datasets (RDD)”, explaining that RDD is the underlying architecture that Spark implements.

Key quote from the paper: “We built a product based on an in‑memory distributed architecture called a resilient distributed dataset, and we named the product Spark.”

Why Spark Was Created – Limitations of Hadoop/MapReduce

MapReduce, derived from Google’s papers, suffers from heavy disk I/O because intermediate results are written to disk, making iterative algorithms (e.g., PageRank, K‑means, logistic regression) and interactive analytics slow.

Spark addresses these pain points by keeping data in memory, enabling fast iteration and interactive queries.

MapReduce Workflow

MapReduce processes data in a single pass:

RecordReader → Mapper → Partitioner → Shuffle & Sort → Reducer → HDFS

. The intermediate data is persisted to disk, preventing reuse in subsequent iterations.

Performance Comparison

Benchmarks show Spark’s first iteration is comparable to MapReduce (e.g., 46 s vs 80 s), but subsequent iterations are dramatically faster (e.g., 3 s vs 76 s) because Spark reuses cached data in memory.

Resilient Distributed Dataset (RDD)

RDD is a fault‑tolerant, in‑memory collection of data partitions. If a node fails, the lost partition can be recomputed from the original data source (e.g., HDFS).

RDDs can be created in three ways:

From external files or a directory of files.

From data already in memory.

From another RDD via transformations.

Interactive Spark Shells

The speaker demonstrated launching pyspark on a virtual machine with Spark 2.1.0, noting that Spark supports Scala, Java, Python, and R, and can be used interactively.

Functional Programming in Spark

Spark leverages functional programming concepts (first‑class functions, anonymous functions) available in Scala, Java 8+, and Python, which simplify distributed data processing compared to the verbose MapReduce code.

Spark SQL and DataFrames

Spark SQL, along with GraphX, Streaming, and MLlib, forms Spark’s “four pillars.” It introduces the DataFrame abstraction (a schema‑aware RDD) and the Catalyst optimizer for query planning.

DataFrames provide a tabular view of data, allowing SQL‑like operations while retaining Spark’s distributed execution engine.

Conclusion

Spark’s in‑memory RDD model, functional API, and SQL integration make it a versatile successor to MapReduce for both batch and interactive workloads, especially when handling iterative machine‑learning algorithms and mixed‑type data.