BestHub
Sign in
Tagged articles

Structured Streaming

11 articles · Page 1 of 1
Cloud Architecture
Cloud Architecture
Apr 14, 2026 · Big Data

Spark SQL Deep Dive: From API to Core for Real‑Time Billion‑Row Processing

This article provides a comprehensive, step‑by‑step guide to mastering Spark SQL in high‑concurrency, billion‑row scenarios, covering the execution chain, architecture layers, Delta Lake integration, performance tuning, production‑grade streaming and batch pipelines, Kubernetes deployment, parameter management, observability, and real‑world case studies.

Delta LakeKubernetesSpark SQL
0 likes · 44 min read
Spark SQL Deep Dive: From API to Core for Real‑Time Billion‑Row Processing
Big Data Technology Tribe
Big Data Technology Tribe
Mar 8, 2026 · Big Data

How Spark Structured Streaming’s Real-Time Mode Achieves Millisecond Latency

This article explains Spark Structured Streaming’s new Real-Time Mode introduced in Spark 4.1, detailing how it reduces latency to the millisecond level by redesigning micro‑batch processing, concurrent stage scheduling, streaming shuffle, and checkpointing, and compares it with Flink’s native streaming.

Real-Time ModeStreamingStructured Streaming
0 likes · 11 min read
How Spark Structured Streaming’s Real-Time Mode Achieves Millisecond Latency
Data Integration and Governance
Data Integration and Governance
Jan 20, 2026 · Big Data

Why Spark Isn't Enough: When to Use Other Tools in Your Data Projects

Although Spark excels at large‑scale, complex computations and offers high‑level APIs like DataFrame and SQL, real‑world projects often require additional tools for data ingestion, task scheduling, and simple streaming, so combining Spark with a data‑integration platform such as FineDataLink yields a more efficient and maintainable pipeline.

Data IntegrationData PipelineFineDataLink
0 likes · 9 min read
Why Spark Isn't Enough: When to Use Other Tools in Your Data Projects
Big Data Technology & Architecture
Big Data Technology & Architecture
Jan 2, 2020 · Big Data

Structured Streaming: Design, Challenges, Programming Model, and Performance Evaluation

This article provides a comprehensive overview of Apache Spark Structured Streaming, describing its declarative API, the challenges of stream processing, the programming model with code examples, query planning, execution modes, production use cases, and performance benchmarks compared with other streaming systems.

SparkStreamingStructured Streaming
0 likes · 42 min read
Structured Streaming: Design, Challenges, Programming Model, and Performance Evaluation
Big Data Technology & Architecture
Big Data Technology & Architecture
Jun 19, 2019 · Big Data

Understanding Spark Structured Streaming StateStore: Architecture, Operations, and Fault Recovery

This article explains the design and implementation of Spark Structured Streaming's StateStore module, covering its distributed architecture, state sharding, versioning, batch read/write, migration, update/query APIs, maintenance compaction, and fault‑tolerance mechanisms that enable incremental continuous queries with exactly‑once guarantees.

SparkStateStoreStreaming
0 likes · 8 min read
Understanding Spark Structured Streaming StateStore: Architecture, Operations, and Fault Recovery
Big Data Technology & Architecture
Big Data Technology & Architecture
Jun 5, 2019 · Big Data

Real-Time Advertising Click Counting with Spark Structured Streaming and Redis Streams

This article presents a complete solution for real‑time advertising click counting using Spark Structured Streaming combined with Redis Streams, detailing the business scenario, data flow, input/output formats, and step‑by‑step implementation including data extraction, processing, storage, and query via Spark‑SQL.

Redis StreamScalaSpark
0 likes · 11 min read
Real-Time Advertising Click Counting with Spark Structured Streaming and Redis Streams
Big Data Technology & Architecture
Big Data Technology & Architecture
Jan 1, 2019 · Big Data

Insights from the Real-Time Big Data Meetup: Spark Structured Streaming Overview

The meetup on September 8, co‑hosted by InfoQ and Huawei Cloud, featured Databricks engineer Tathagata Das explaining Spark Structured Streaming’s concepts, fault‑tolerance, performance, event‑time handling, and real‑world use cases such as Apple’s security platform, highlighting its scalability and integration with various data sources.

SparkStructured Streamingbig data
0 likes · 8 min read
Insights from the Real-Time Big Data Meetup: Spark Structured Streaming Overview
High Availability Architecture
High Availability Architecture
May 19, 2016 · Big Data

Comprehensive Overview of Apache Spark: Architecture, RDD Principles, Execution Modes, and Spark 2.0 Features

This article provides an in‑depth technical overview of Apache Spark, covering its core concepts such as RDDs, transformation and action operations, execution models, Spark 2.0 enhancements like unified DataFrames/Datasets, whole‑stage code generation, Structured Streaming, and practical performance‑tuning guidance.

DataFramesPerformance OptimizationRDD
0 likes · 20 min read
Comprehensive Overview of Apache Spark: Architecture, RDD Principles, Execution Modes, and Spark 2.0 Features