This article explains Hive's partitioning and bucketing techniques, compares their purposes, advantages, and pitfalls, and shows how to combine them with concrete SQL examples to improve query performance, reduce I/O, and optimize joins and sampling in large data warehouses.
This article explains how Spark ≥ 3.3 introduces Storage Partitioned Join (SPJ) to avoid costly shuffle operations when joining partitioned V2 source tables such as Apache Iceberg, detailing the required conditions, configuration settings, practical code examples, and various join scenarios including mismatched partitions and data skew.
This article explains Doris’s prefix index mechanism, best practices for partitioning and bucketing, and how to correctly use Flink’s batch writes with sequence columns to ensure ordered updates, providing practical guidance for optimizing OLAP workloads in Apache Doris.
This article explains how to model massive time‑series data in Cassandra by using bucketing techniques to control partition size, avoid hotspots, and improve write and read performance, including practical CQL schema examples and Python code for concurrent queries.
Effective ETL optimization is crucial for data warehouse construction, and this guide outlines three core strategies—ensuring proper resource configuration, leveraging data characteristics for optimal partitioning and bucketing, and monitoring task execution—providing practical principles, pitfalls, and case studies to maximize ETL efficiency.
