Why Spark Beats Hadoop: Exploring RDDs, In‑Memory Computing, and Fault Tolerance

Overview

Apache Spark is an open‑source, general‑purpose parallel computing framework originally developed at UC Berkeley. It implements a MapReduce‑like model but stores intermediate results in memory, which greatly speeds up iterative algorithms used in data mining and machine learning.

Spark vs. Hadoop

Key differences include:

Intermediate data resides in memory, providing higher efficiency for iterative workloads.

Spark offers a richer set of operations (transformations such as map, filter, flatMap, groupByKey and actions like count, collect, save).

It supports multiple languages (Scala, Java, Python) and interactive shells.

However, Spark is not suited for fine‑grained asynchronous updates like incremental web crawlers.

Spark Ecosystem

Shark (Hive on Spark) provides HiveQL compatibility.

Spark Streaming processes data streams in micro‑batches, enabling both batch and real‑time analytics.

Bagel implements Pregel‑style graph processing, e.g., PageRank.

Core Concept: Resilient Distributed Dataset (RDD)

An RDD is Spark’s fundamental abstraction representing an immutable, partitioned collection of elements that can be operated on in parallel.

Features: immutable, partitioned, serializable, can be cached in memory.

Benefits: lineage tracking enables automatic recomputation of lost partitions.

RDD operations are classified as:

Transformations (lazy) – create a new RDD (e.g., map, filter, join).

Actions (eager) – trigger computation and return results (e.g., count, collect, saveAsTextFile).

Example of Transformations and Actions

val sc = new SparkContext("local", "Example")
val rddA = sc.textFile("hdfs://...")
val rddB = rddA.flatMap(line => line.split("\\s+")).map(word => (word, 1))
val rddC = rddB.reduceByKey(_ + _)
rddC.saveAsTextFile("hdfs://.../output")

Lineage and Fault Tolerance

Each RDD records its lineage – the sequence of transformations that produced it. When a partition is lost, Spark recomputes it from the lineage, avoiding fine‑grained logging. Dependencies are classified as:

Narrow dependencies : each child partition depends on at most one parent partition.

Wide dependencies : a child partition depends on many parent partitions, requiring shuffles.

Fault tolerance is achieved via checkpointing (data or update logs) and lineage reconstruction.

Resource Management and Scheduling

Spark can run on Standalone, Apache Mesos, or Hadoop YARN. YARN integration allows Spark to share cluster resources with Hadoop jobs.

Programming Interfaces

Scala

Spark is written in Scala; the interactive SparkShell lets users experiment quickly.

val sc = new SparkContext("local", "App")
val lines = sc.textFile("hdfs://.../data")
val words = lines.flatMap(_.split("\\s+")).filter(_.startsWith("spar"))
val top5 = words.take(5)

Java

Java programs use the JavaSparkContext wrapper.

JavaSparkContext sc = new JavaSparkContext(new SparkConf().setAppName("App"));
JavaRDD<String> lines = sc.textFile("hdfs://.../data");
JavaRDD<String> words = lines.flatMap(line -> Arrays.asList(line.split(" ")).iterator());

Python

PySpark provides a Python API via Py4J.

from pyspark import SparkContext
sc = SparkContext("local", "App")
words = sc.textFile("/usr/share/dict/words").filter(lambda w: w.startswith("spar")).take(5)

Usage Examples

Standalone Mode

Deploy a Spark master and workers, configure SPARK_MASTER_IP, SPARK_WORKER_CORES, etc., then start the cluster with ./sbin/start-all.sh. Users can submit jobs via spark-submit or use the interactive shell.

YARN Mode

Package Spark with sbt package assembly, then submit with

./bin/run-deploy.yarn.Client --class org.apache.spark.examples.SparkPi --master yarn-client --jar examples/target/scala-2.9.3/spark-examples-assembly-0.8.0-SNAPSHOT.jar

.

Spark Shell

After starting a Standalone cluster, launch $SPARK_HOME/spark-shell. The shell provides a pre‑created sc (SparkContext) for interactive experimentation.

Driver Program

A full driver program creates its own SparkContext, reads input from HDFS, performs transformations, and writes output back to HDFS.

object WordCount {
  def main(args: Array[String]) {
    if (args.length == 0) { println("usage: WordCount <master>"); return }
    val hdfsPath = "hdfs://hadoop1:8020"
    val sc = new SparkContext(args(0), "WordCount", System.getenv("SPARK_HOME"), Seq())
    val text = sc.textFile(hdfsPath + "/" + args(1))
    val result = text.flatMap(line => line.split("\\s+"))
                     .map(word => (word, 1))
                     .reduceByKey(_ + _)
    result.saveAsTextFile(hdfsPath + "/" + args(2))
  }
}