Understanding Spark SQL Analyzer: Principles, Optimization Cases, and Rule‑Pruning in Spark 3.2+

The fourth session of the Spark series focuses on the Spark SQL analysis layer, reviewing previous topics and outlining five parts: a recap, analysis layer principles, an optimization case, a summary, and Q&A.

It introduces the role of the Analyzer, which binds the abstract syntax tree produced by the parser to metadata (catalogs, tables, functions) using components like SessionCatalog and CatalogManager , and explains how built‑in and user‑defined functions are resolved via FunctionRegistry .

The article describes how analysis rules are applied through a rule executor, with examples such as ResolveRelations that locate UnresolvedRelation nodes and resolve them to tables, views, or other data sources.

A key optimization case is presented: before Spark 3.2, rule traversal used exhaustive depth‑first or breadth‑first methods ( resolveOperatorsDown , resolveOperatorsUp ), causing unnecessary CPU consumption. Spark 3.2 introduces tree‑pruning via methods like resolveOperatorsDownWithPruning and resolveOperatorsUpWithPruning , which evaluate TreePatternBits to apply rules only when relevant patterns are present.

The pruning mechanism relies on three components: TreePattern (an enum of pattern types), TreePatternBits (a bit‑set for fast pattern matching), and transformed functions with pruning support (e.g., transformExpressionsUpWithPruning ).

As an illustrative rule, RevolveRandomSeed shows how the new pruning APIs are used to skip processing when the logical plan does not contain the EXPRESSION_WITH_RANDOM_SEED pattern.

The session concludes with a summary, references to the “Stop earlier without traversing the entire tree” article, and a Q&A covering monitoring of ResolveRelations, user perception of logical‑plan optimizations, Spark vs. Flink resource trade‑offs, and additional logical‑plan optimizations.