Essential Spark Interview Q&A: Master Data Warehouse Engineer Questions

Spark Interview Questions and Answers

Question: Usually Spark runs faster than MapReduce. Explain which built‑in mechanisms give Spark this advantage. Spark improves performance mainly by keeping intermediate shuffle data in memory, using the resilient distributed dataset (RDD) lineage for efficient fault recovery, offering a rich set of transformation and action APIs, and providing a DAG scheduler that optimizes stage execution. Additionally, Spark’s executor model runs many tasks in parallel, and memory management separates storage and execution memory for better utilization.

Question: Hadoop and Spark usage scenarios? Both are suited for offline batch analytics, but Hadoop excels when a single job processes extremely large data volumes, whereas Spark is preferred for iterative workloads, machine‑learning tasks, and moderate‑size data (e.g., 80 GB compressed, 10‑node clusters) where it can finish in minutes instead of longer MapReduce runs.

Question: How does Spark ensure rapid recovery after a master node failure? Deploy a standby Spark master and use a monitoring shell script to detect master health; upon failure, the script automatically restarts the master.

Question: Similarities and differences between Hadoop and Spark? Both are distributed computing frameworks, but Hadoop’s MapReduce offers only map and reduce operations and writes intermediate data to HDFS, leading to higher I/O latency. Spark runs in‑memory, provides many operators (map, filter, join, etc.), supports streaming and graph processing, and uses a DAG scheduler for flexible stage planning.

Question: RDD persistence principle? RDDs can be persisted in memory using cache() (which calls persist(StorageLevel.MEMORY_ONLY) ) or with other storage levels via persist() . To remove cached data, call unpersist() .

Question: What is the checkpoint mechanism? Checkpointing writes RDDs to a reliable file system (e.g., HDFS) after a job finishes, creating a new lineage‑independent RDD. It is useful for long‑running, complex Spark applications to limit recomputation and provide driver fault tolerance in Spark Streaming.

Question: Difference between checkpoint and persistence? Persistence stores data only in the BlockManager (memory or disk) and retains the original lineage, while checkpoint creates a new RDD without lineage and stores data in a fault‑tolerant file system, reducing the risk of data loss.

Question: Do you understand the RDD mechanism? RDD (Resilient Distributed Dataset) is Spark’s core abstraction: an immutable, partitioned collection of records stored across cluster nodes. Transformations build a lineage graph that enables automatic recomputation of lost partitions. RDDs can be cached in memory and spill to disk when needed.

Question: Spark Streaming and its basic working principle? Spark Streaming extends the core API to process real‑time data streams. Input data (e.g., from Kafka, Flume) is divided into micro‑batches; each batch is treated as an RDD and processed by the Spark engine, producing a continuous output stream.

Question: DStream and its basic working principle? DStream is a high‑level abstraction representing a continuous stream of data. Internally it generates a sequence of RDDs, each corresponding to a time interval, and supports operations like map, reduce, join, and window.

Question: What components does Spark have? Master (cluster manager), Worker (execution node), Driver (runs the main program and creates SparkContext), SparkContext (coordinates the application lifecycle), and Client (submission entry point).

Question: How does Spark work internally? When a client submits a job, the Driver creates a SparkContext, builds a DAG of transformations, the DAGScheduler divides the DAG into stages, and the TaskScheduler distributes tasks to executors on workers for execution.

Question: Explain wide and narrow dependencies. Wide dependency (shuffle) occurs when a parent RDD partition’s data may be sent to many child partitions. Narrow dependency is a one‑to‑one relationship where each child partition depends on a single parent partition.

Question: Do you know Spark master‑standby HA mechanism? In standalone mode, Spark can configure two masters. If the active master fails, the standby master takes over. HA can be file‑system based (manual switch) or ZooKeeper‑based (automatic failover).

Question: Which Hadoop problems does Spark solve? Spark provides a richer API, in‑memory computation, iterative processing, lower latency, and supports streaming, reducing the need for multiple MapReduce jobs and improving resource utilization.

Question: Causes and solutions for data skew? Data skew occurs when a few partitions hold disproportionately large amounts of data, slowing down tasks. Mitigation includes choosing appropriate keys, adding a salt or custom partitioner, and avoiding shuffle‑heavy operators like groupByKey , reduceByKey , join , etc.

Question: When using SparkSQL, do you prefer DataFrames or raw SQL? Why? DataFrames offer a higher‑level, optimized API with automatic catalyst optimization, making them easier to use for most cases, while raw SQL can be useful for complex queries that are more naturally expressed in SQL syntax.

Question: Write a practical interview coding task. Given an HDFS file where each line contains work_id,user_id,gender , count distinct male and female users per work ID and output work_id, male_count, female_count .

sc.textFile("path")
  .map(line => line.split(","))
  .map(arr => ((arr(0), arr(2)), 1))
  .reduceByKey(_ + _)
  .map{ case ((workId, gender), cnt) => (workId, gender, cnt) }

Question: Which is faster, reduceByKey or groupByKey, and why? reduceByKey performs a local combine before shuffling, reducing data transfer and improving speed, whereas groupByKey shuffles all values, leading to higher network I/O and possible OOM errors.

Question: Why does Spark master HA not affect already running jobs? Running jobs have already acquired resources and maintain communication between Driver and Executors; they do not depend on the master after submission.

Question: What data does Spark store in ZooKeeper for HA? Spark uses the configuration spark.deploy.zookeeper.dir to persist metadata such as Worker, Driver, Application, and Executor information, enabling the standby master to recover the cluster state.

1. All running programs continue unchanged during master switch.
2. New job submissions are blocked until the new active master is ready.