Tagged articles
DataFusion
10 articles · Page 1 of 1
Apache Auron graduates from incubation to become a top-level project after 13 months, delivering 2x faster Spark query performance and 50% resource savings via its Rust-native vectorized execution engine, already proven at exabyte scale across companies like Kuaishou, DiDi, and LinkedIn.
Apache ArrowApache AuronApache Spark
0 likes · 14 min read

This article explains the merge‑insert operation in Lance, detailing its SQL‑like semantics, typical use‑cases such as bulk upserts and conditional deletes, the underlying DataFusion planning and execution flow, the generation of the __action column, and the handling of source‑side duplicate rows.
DataFusionLanceRust
0 likes · 7 min read

Lenovo transformed its traditional marketing technology workflow by deploying self‑developed AI agents and a unified data‑fusion framework, tackling high content‑creation costs, fragmented channel deployment, and manual data analysis, and achieved measurable efficiency, cost, and performance gains across the enterprise.
AIDataFusionIntelligentAgents
0 likes · 21 min read

This article explains DataFusion, a Rust‑written, Arrow‑based query engine that offers high performance, extensibility, and seamless integration with various data sources, detailing its architecture, execution model, Rust advantages, and practical usage examples for building modern data‑warehouse solutions.
Apache ArrowDataFusionQuery Engine
0 likes · 15 min read

Blaze is a Rust‑implemented, DataFusion‑based vectorized execution engine created by Kuaishou to accelerate Spark SQL queries, delivering up to 60% faster computation, 30% average compute‑power gains in production, and extensive architectural innovations such as native engine, protobuf protocol, JNI bridge, and Spark extension, while being open‑source and compatible with Spark 3.0‑3.5.
DataFusionRustSpark
0 likes · 11 min read

This article introduces DataFusion, a high‑performance, Rust‑based query engine that leverages Apache Arrow’s columnar memory format to enable fast, extensible data processing across multiple storage formats and cloud sources, explains its architecture, execution model, and provides practical Rust code examples for custom extensions.
Apache ArrowDataFusionQuery Engine
0 likes · 16 min read

Blaze is Kuaishou's self‑developed native execution engine that leverages Rust, DataFusion, and SIMD vectorization to accelerate Spark workloads, offering a 30%+ compute boost, detailed architectural components, deep production‑grade optimizations, and a roadmap for broader adoption.
DataFusionNative ExecutionRust
0 likes · 13 min read

This article explains why new data systems are needed, introduces Apache Arrow and its columnar in‑memory format, describes dynamic read‑time modeling, outlines the system’s execution flow, storage and indexing strategies, and shares practical tips and extensions for building scalable big‑data solutions.
AceroApache ArrowData Systems
0 likes · 20 min read

This article explains why new data systems are needed, introduces Apache Arrow’s columnar in‑memory format and its zero‑copy advantages, describes how to model data at read time, outlines the execution flow with Acero and SQL planning, and shares practical tips and extensions for building robust, dynamic‑schema data platforms.
AceroApache ArrowData Systems
0 likes · 20 min read

This article presents Blaze, a Kuaishou‑built native execution middleware for SparkSQL that leverages Apache DataFusion to achieve vectorized operator execution, detailing its architecture, implementation, performance gains, current coverage, benchmark results, production rollout, and future development plans.
DataFusionNative ExecutionSpark
0 likes · 17 min read
