BestHub
Sign in
Tagged articles

adaptive query execution

12 articles · Page 1 of 1
Java Architect Handbook
Java Architect Handbook
Jul 29, 2026 · Big Data

What Is Data Skew in Distributed Computing, Its Impact, and How to Fix It

The article defines data skew as an uneven key distribution after shuffle that makes a few tasks handle most of the data, explains the resulting slowdown, OOM, low resource utilization and job failure, and then details diagnosis methods and concrete solutions for group‑by, join, null‑value and framework‑level scenarios.

Data SkewDistributed ComputingHive
0 likes · 15 min read
What Is Data Skew in Distributed Computing, Its Impact, and How to Fix It
Big Data Technology & Architecture
Big Data Technology & Architecture
Nov 12, 2024 · Big Data

Adaptive Query Execution (AQE) in Apache Spark 4.0: A Revolution in Query Optimization

This article explains how Adaptive Query Execution (AQE) in Apache Spark 4.0 dynamically optimizes query plans through features such as join reordering, partition pruning, skew handling and coalescing, delivering significant performance gains, resource efficiency and reduced manual tuning across real‑world big‑data workloads.

Apache SparkBig DataSpark 4.0
0 likes · 13 min read
Adaptive Query Execution (AQE) in Apache Spark 4.0: A Revolution in Query Optimization
DataFunSummit
DataFunSummit
Sep 27, 2022 · Big Data

Apache Spark Adaptive Query Execution and Kyuubi Optimization Practices for Data Warehousing

This article presents a detailed overview of Apache Spark's Adaptive Query Execution evolution, its optimization techniques, and performance gains, followed by an in‑depth discussion of Apache Kyuubi's architecture, security integrations, cloud‑native capabilities, and practical Rebalance + Z‑Order strategies that enhance data‑warehouse task efficiency and query performance.

Apache SparkBig Data OptimizationKyuubi
0 likes · 19 min read
Apache Spark Adaptive Query Execution and Kyuubi Optimization Practices for Data Warehousing
JD Retail Technology
JD Retail Technology
Jan 27, 2022 · Big Data

How JD’s Custom Spark Engine Tackles Data Skew for Massive Offline Jobs

This article explains JD’s self‑developed data‑skew mitigation solution for Spark, detailing the problem of uneven key distribution, the limitations of the open‑source AQE implementation, and JD’s OptimizeSkewedJoinV2 algorithm that dramatically reduces stage latency in large‑scale join workloads.

Big DataData SkewDistributed Computing
0 likes · 13 min read
How JD’s Custom Spark Engine Tackles Data Skew for Massive Offline Jobs
Big Data Technology & Architecture
Big Data Technology & Architecture
Dec 23, 2021 · Big Data

Key Spark Configuration Parameters and Their Explanations

This article presents a comprehensive list of essential Spark configuration settings—including executor memory, off‑heap memory, memory fractions, shuffle options, and adaptive query execution parameters—each accompanied by a concise description to help users fine‑tune Spark performance.

Big DataMemory ManagementShuffle
0 likes · 6 min read
Key Spark Configuration Parameters and Their Explanations
Big Data Technology & Architecture
Big Data Technology & Architecture
Dec 21, 2021 · Big Data

Understanding Spark 3.0 Adaptive Query Execution (AQE) and Dynamic Partition Pruning (DPP)

This article explains the two most important Spark 3.0 features—Adaptive Query Execution and Dynamic Partition Pruning—detailing how AQE dynamically optimizes join strategies, partition coalescing, and skew handling, while DPP reduces I/O by pruning irrelevant fact‑table partitions at runtime.

Big DataDynamic Partition PruningSQL Optimization
0 likes · 10 min read
Understanding Spark 3.0 Adaptive Query Execution (AQE) and Dynamic Partition Pruning (DPP)
Big Data Technology Architecture
Big Data Technology Architecture
Nov 16, 2021 · Big Data

Understanding Adaptive Query Execution and Dynamic Partition Pruning in Apache Spark 3.0

This article explains how Apache Spark 3.0 improves SQL workload performance through Adaptive Query Execution (AQE) and Dynamic Partition Pruning (DPP), detailing their design principles, runtime optimizations, configuration parameters, and practical examples that demonstrate reduced shuffle partitions, smarter join strategies, and handling of data skew.

Dynamic Partition PruningSQL OptimizationSpark
0 likes · 9 min read
Understanding Adaptive Query Execution and Dynamic Partition Pruning in Apache Spark 3.0
Big Data Technology Architecture
Big Data Technology Architecture
Aug 12, 2020 · Big Data

Overview of New Features and Improvements in Apache Spark 3.0

Apache Spark 3.0 introduces a suite of performance enhancements, richer APIs, improved monitoring, SQL compatibility, new data sources, and ecosystem extensions, including Adaptive Query Execution, Dynamic Partition Pruning, Join Hints, pandas UDF improvements, and accelerator‑aware scheduling, to boost scalability and ease of use for big‑data workloads.

Apache SparkData EngineeringSpark 3.0
0 likes · 15 min read
Overview of New Features and Improvements in Apache Spark 3.0
Big Data Technology Architecture
Big Data Technology Architecture
Aug 8, 2020 · Big Data

Overview of SQL Performance Improvements in Apache Spark 3.0

Apache Spark 3.0 introduces extensive SQL performance enhancements, including a new explain format, expanded join hints, adaptive query execution, dynamic partition pruning, enhanced nested column pruning, improved aggregation code generation, and support for newer Scala and Java versions, all aimed at optimizing query planning and execution.

Apache SparkSQL Optimizationadaptive query execution
0 likes · 14 min read
Overview of SQL Performance Improvements in Apache Spark 3.0
Big Data Technology Architecture
Big Data Technology Architecture
Jun 20, 2020 · Big Data

Apache Spark 3.0.0 Release: New Features, Improvements, and Timeline

Apache Spark 3.0.0, released after a 21‑month development cycle and several preview and release‑candidate votes, introduces major enhancements such as Dynamic Partition Pruning, Adaptive Query Execution, accelerator‑aware scheduling, DataSource V2, expanded pandas UDFs, new join hints, richer monitoring, SparkR vectorization, Kafka header support, and broader ecosystem integrations, while fixing over 3,400 issues.

Apache SparkDataSource V2Dynamic Partition Pruning
0 likes · 17 min read
Apache Spark 3.0.0 Release: New Features, Improvements, and Timeline
dbaplus Community
dbaplus Community
Jun 20, 2020 · Big Data

What’s New in Apache Spark 3.0? Explore Dynamic Partition Pruning, AQE, and More

Apache Spark 3.0, released after a 21‑month development cycle, introduces dynamic partition pruning, adaptive query execution, accelerator‑aware scheduling, DataSource V2, enhanced pandas UDFs, new join hints, richer monitoring, ANSI‑SQL compatibility, SparkR vectorization, Kafka header support, and numerous platform upgrades, all backed by over 3,400 resolved issues.

Apache SparkBig DataDataSource V2
0 likes · 17 min read
What’s New in Apache Spark 3.0? Explore Dynamic Partition Pruning, AQE, and More