Overview of Real-Time Big Data Processing: Spark Structured Streaming, CarbonData, Flink, and Cloud Stream

The demand for millisecond‑to‑second real‑time analytics is rising across enterprises, prompting the adoption of high‑performance big‑data platforms such as Spark Structured Streaming, CarbonData, and Flink.

Spark Structured Streaming offers unified APIs, fault‑tolerant checkpointing, micro‑batch and continuous processing, support for multiple sources (Kafka, Files, Kinesis), and exactly‑once semantics, making it suitable for complex event‑time workloads.

CarbonData is a high‑performance columnar storage solution that combines fast filtering, multi‑level indexing, dictionary encoding, and pre‑aggregation to enable trillion‑row analytics with sub‑second response times. It integrates tightly with SparkSQL, supports segment‑level management, and provides DataMap indexes for accelerated queries.

Flink at Meituan powers a massive real‑time platform handling trillions of events daily. The architecture leverages Kafka for ingestion, Flink for low‑latency processing, and a layered resource‑isolation strategy on YARN. Key practices include HA JobManager deployment, checkpoint‑based recovery, multi‑zone redundancy, and custom retry logic for Kafka I/O.

Meituan’s use cases span the Petra real‑time metric aggregation system and the MLX machine‑learning platform, both of which exploit Flink’s event‑time processing, state management, and exactly‑once guarantees.

Comparison and Cloud Stream – Huawei’s Cloud Stream Service unifies Flink and Spark (Streaming & Structured Streaming) under a serverless, multi‑tenant model. It addresses limitations of Spark Streaming, offers visual StreamSQL editing, supports a rich ecosystem (Kafka, Hadoop, Elasticsearch, etc.), and delivers millisecond‑level latency with million‑message‑per‑second throughput.

Overall, the article highlights the evolution of streaming technologies, their architectural trade‑offs, and emerging trends such as unified APIs, state‑ful processing, and cloud‑native deployment models.