Inside StarRocks Optimizer: Architecture, Multi‑Stage Optimization, and Advanced Features

This article provides a comprehensive technical overview of StarRocks' query optimizer, covering its evolution, core architecture, multi‑stage optimization pipeline, key optimizations such as multi‑join colocate, low‑cardinality global dictionary, MV union rewrite, and advanced mechanisms like cost‑estimation fixes, query feedback, adaptive execution, runtime filters, join‑reorder strategies, and SQL plan management.

Adaptive ExecutionMaterialized ViewsOLAP

0 likes · 26 min read

Inside StarRocks Optimizer: Architecture, Multi‑Stage Optimization, and Advanced Features

DataFunTalk

Nov 12, 2023 · Big Data

MaxCompute Incremental Update Architecture, Intelligent Materialized Views, and Adaptive Execution Optimizations

This article presents a comprehensive overview of MaxCompute's near‑real‑time incremental update and processing architecture, the design of Transactional Table 2.0, intelligent materialized view evolution and recommendation, as well as multi‑level adaptive execution optimizations for the SQL engine, illustrating how these innovations improve efficiency, cost, and scalability for large‑scale data workloads.

Adaptive ExecutionMaxComputeSQL Engine

0 likes · 20 min read

MaxCompute Incremental Update Architecture, Intelligent Materialized Views, and Adaptive Execution Optimizations

JD Retail Technology

Apr 14, 2023 · Big Data

Understanding Data Skew and Its Mitigation in Hive and Spark

This article explains the concept of data skew, its symptoms such as slow tasks and OOM errors, and provides comprehensive mitigation techniques and configuration examples for Hive and Spark, including custom partitioning, map joins, adaptive execution, and key detection methods.

Adaptive ExecutionBig DataData Skew

0 likes · 15 min read

Understanding Data Skew and Its Mitigation in Hive and Spark

Big Data Technology Architecture

Apr 1, 2021 · Big Data

Spark Adaptive Execution: Dynamic Shuffle Partition, Broadcast Join, and Skew Handling

The article explains the limitations of static shuffle partitions, execution‑plan estimation, and data skew in Spark SQL, and describes how Spark Adaptive Execution can automatically adjust shuffle partition numbers, switch join strategies, and mitigate skew through configurable parameters and code examples.

Adaptive ExecutionBroadcast JoinData Skew

0 likes · 11 min read

Spark Adaptive Execution: Dynamic Shuffle Partition, Broadcast Join, and Skew Handling

Big Data Technology & Architecture

Jul 1, 2020 · Big Data

Overview of Spark SQL Adaptive Execution Optimization Engine

This article explains Spark SQL's Adaptive Execution engine, covering its background, dynamic plan adjustments, shuffle partition tuning, data skew mitigation techniques, and the key configuration parameters needed to enable and fine‑tune adaptive query execution for improved performance.

Adaptive ExecutionBig DataConfiguration

0 likes · 7 min read

Overview of Spark SQL Adaptive Execution Optimization Engine