Understanding Spark Streaming Integration with Kafka: Receiver-based and Direct Approaches

Spark Streaming (SS) is a high‑throughput, fault‑tolerant stream processing framework built on Spark that can ingest data from various sources such as Kafka, Flume, Twitter, ZeroMQ, Kinesis, and TCP sockets. It divides the continuous data stream into micro‑batches (RDDs) based on a configurable batch interval, applies transformations like map, reduce, join, and window, and then executes the computation on the Spark engine.

The fundamental abstraction in SS is the Discretized Stream (DStream), which represents a sequence of RDDs indexed by time. Windowing concepts include the window length (the time span over which data is aggregated) and the slide interval (how often the window moves), both of which must be integer multiples of the batch interval.

SS provides two ways to consume data from Kafka:

Receiver‑based Approach – uses Kafka’s high‑level consumer API via Receivers that store incoming data in Spark executors before processing. To achieve zero data loss, Write‑Ahead Logs (WAL) can be enabled, persisting received data to HDFS for recovery.

Direct Approach (No Receivers) – periodically queries Kafka for the latest offsets and creates a KafkaRDD for each batch, eliminating the need for receivers and WAL, offering higher efficiency and exactly‑once semantics when combined with checkpointing. Key differences between the two approaches:

Receiver‑based incurs additional memory overhead (currentBuffer, blocksForPushing, blockPushingThread) and may cause OOM or GC pauses; Direct Approach pulls data on‑demand, avoiding buffer buildup.

Direct Approach aligns Spark partitions with Kafka partitions, simplifying parallelism and improving performance.

Direct Approach relies on checkpointing to store offsets, guaranteeing at‑least‑once delivery; exactly‑once requires idempotent processing or transactional logic.

Configuration tips include setting spark.streaming.receiver.maxRate to limit receiver ingestion rate, adjusting spark.streaming.blockInterval and spark.streaming.blockQueueSize to control memory usage, and enabling spark.streaming.backpressure.enabled for dynamic rate adaptation. Checkpointing in SS serializes the DStream graph and offset information without persisting actual data, keeping the checkpoint size small (tens of KB). The checkpoint data includes the batch timestamp and Kafka offset metadata, enabling recovery after failures. Overall, based on practical experience, the Direct Approach combined with checkpointing provides greater stability, simpler configuration, and better performance for Spark‑Kafka integration, while the Receiver‑based method may still be useful when specific legacy requirements exist.