Why Spark Overtook MapReduce: Core Advantages and RDD Programming Model

1 Spark’s Rise Over MapReduce

Apache Spark was created by the AMP Lab at UC Berkeley. Within two years it captured the dominant share of the big‑data processing market, largely because it executes jobs up to 100× faster than classic MapReduce and offers a concise, high‑level API.

2 Advantages of Spark

In‑memory computation reduces I/O latency, giving higher throughput.

Unified programming model (batch, interactive, streaming, machine learning) under a single engine.

Native integration with YARN, HDFS, and other Hadoop components simplifies migration.

Rich libraries (Spark SQL, Structured Streaming, MLlib, GraphX) extend functionality without additional frameworks.

3 Spark Programming Model – RDD

Resilient Distributed Dataset (RDD) is the fundamental abstraction. An RDD represents an immutable, partitioned collection of records that can be operated on in parallel. Transformations create a new RDD from an existing one; actions trigger the actual computation.

3.1 Core Transformations (return an RDD)

map(func)

– apply func to each element. filter(func) – keep elements where func returns true. union(other) – concatenate two RDDs. reduceByKey(func, [numPartitions]) – aggregate values with the same key. join(other, [numPartitions]) – inner join on keys. groupByKey([numPartitions]) – group values by key.

… (e.g., flatMap, distinct, sample)

3.2 Actions (materialize a result)

collect()

– return all elements to the driver. count() – number of elements. take(n) – retrieve the first n elements. saveAsTextFile(path) – write RDD to HDFS or local FS. reduce(func) – aggregate all elements using func.

Transformations are evaluated lazily: Spark builds a logical DAG of RDD operations. The DAG is optimized and only executed when an action is called, which minimizes data shuffling and enables pipelining.

3.3 Example: WordCount in Scala

val text = sc.textFile("hdfs://path/input")
val counts = text.flatMap(_.split("\\s+"))
                 .map(word => (word, 1))
                 .reduceByKey(_ + _)
counts.saveAsTextFile("hdfs://path/output")

The three logical steps are:

Split each line into words.

Map each word to a (word, 1) pair.

Aggregate pairs by key to compute the total count.

4 Spark Ecosystem

Spark SQL – SQL and DataFrame API for structured data.

Structured Streaming – unified batch‑and‑stream processing.

MLlib – scalable machine‑learning algorithms.

GraphX – graph‑parallel computation.

5 Summary

Spark’s success stems from in‑memory execution, a concise RDD‑based API, and lazy evaluation that together deliver orders‑of‑magnitude speed improvements over MapReduce while preserving compatibility with existing Hadoop storage and resource managers.