Deep Dive into Apache Spark SQL: Concepts, Core Components, and API

Apache Spark SQL originated from the early Spark 0.6 era and has evolved into a mature module for processing structured data, leveraging a layered architecture that transforms SQL text into an Abstract Syntax Tree (AST) composed of TreeNode elements, which are then assembled into a QueryPlan.

The system distinguishes between LogicalPlan (the logical representation of a query) and PhysicalPlan (the executable plan). LogicalPlan is optimized by a series of rules executed by a RuleExecutor, while the GenericStrategy and QueryPlanner collaborate to translate the optimized logical plan into a physical plan ready for execution.

Key core components include:

SparkSqlParser : parses SQL text into an AST using Antlr4, producing structures such as Catalyst Expressions, LogicalPlan, and CatalystIdentifier.

Analyzer : binds the parsed logical plan to metadata from catalogs, producing an Analyzed Logical Plan.

Optimizer : applies performance‑oriented rules (e.g., RewriteDistinctAggregates, AQEOptimizer) to generate an Optimized Logical Plan.

SparkPlanner and SparkStrategy : convert the optimized logical plan into one or more SparkPlan physical operators (e.g., CollectLimitExec, Shuffle, BroadcastJoin).

SQLConf , FunctionRegistry , DataSourceManager , CatalogPlugin , CatalogManager , SessionCatalog : provide configuration, function lookup, data‑source registration, and catalog services.

The Spark Session acts as the entry point, exposing Dataset and DataFrame abstractions; Dataset offers a typed collection while DataFrame provides an untyped row view. APIs such as DataFrameReader, DataFrameWriter, DataFrameWriterV2, MergeIntoWriter, and DataStreamWriter enable loading, persisting, and streaming data to various storage systems.

A Q&A segment addresses practical concerns, including Spark Streaming vs. Flink, data skew mitigation with Adaptive Query Execution, optimization of multiple COUNT(DISTINCT) aggregations, differences between Structured Streaming and legacy Streaming, native execution projects like Gluten, and strategies for handling small files and checkpointing.

The presentation concludes with a brief company introduction (Zhejiang Shuxin Network Co., Ltd.) and information about upcoming events and community engagement.