Tagged articles
Apache Arrow
21 articles · Page 1 of 1
Apache Auron graduates from incubation to become a top-level project after 13 months, delivering 2x faster Spark query performance and 50% resource savings via its Rust-native vectorized execution engine, already proven at exabyte scale across companies like Kuaishou, DiDi, and LinkedIn.
Apache ArrowApache AuronApache Spark
0 likes · 14 min read

The article shows that Polars, a query‑compiling DataFrame library, can accelerate ten‑million‑row GroupBy workloads by 6‑10× compared with Pandas, explains the underlying optimizer, Arrow columnar engine and Rust parallelism, provides a 20‑item syntax map, three real migration scenarios, streaming for out‑of‑memory data, and AI‑pipeline use cases, and offers a step‑by‑step migration guide.
Apache ArrowDataFramesPandas
0 likes · 21 min read

This article breaks down Dremel's columnar encoding for nested data, detailing the definition‑level and repetition‑level concepts, showing step‑by‑step examples of encoding and reconstructing JSON‑like schemas, and explaining the limits of single‑column reconstruction.
Apache ArrowDremelParquet
0 likes · 9 min read

pyarrow.Schema is the Python representation of an Arrow table schema, describing column names, types, nullability, and other metadata, and it is essential for defining, inspecting, serializing, and interfacing data structures across libraries like Pandas, Polars, and Arrow‑based query engines.
Apache ArrowPyArrowPython
0 likes · 4 min read

This article explains how Baidu’s Feed real‑time data warehouse was rebuilt using a pure streaming architecture, detailing the limitations of the previous stream‑batch design, the technical solutions—including core/non‑core data separation, metric calculation in streaming, and Parquet storage with Apache Arrow—and the resulting cost reductions, latency improvements, and future roadmap.
Apache ArrowParquetbatch processing
0 likes · 17 min read

This article explains DataFusion, a Rust‑written, Arrow‑based query engine that offers high performance, extensibility, and seamless integration with various data sources, detailing its architecture, execution model, Rust advantages, and practical usage examples for building modern data‑warehouse solutions.
Apache ArrowDataFusionQuery Engine
0 likes · 15 min read

This article introduces DataFusion, a high‑performance, Rust‑based query engine that leverages Apache Arrow’s columnar memory format to enable fast, extensible data processing across multiple storage formats and cloud sources, explains its architecture, execution model, and provides practical Rust code examples for custom extensions.
Apache ArrowDataFusionQuery Engine
0 likes · 16 min read

The article reviews pandas 2.0’s major upgrades—including an Apache Arrow backend that speeds up CSV reads by over 30×, new Arrow dtypes, a nullable‑numpy dtype for missing values, a copy‑on‑write memory model, optional dependencies, and benchmark comparisons with ydata‑profiling—highlighting the library’s enhanced performance, flexibility, and interoperability for data‑intensive Python workflows.
Apache ArrowPandasPython
0 likes · 15 min read

This article explains why new data systems are needed, introduces Apache Arrow and its columnar in‑memory format, describes dynamic read‑time modeling, outlines the system’s execution flow, storage and indexing strategies, and shares practical tips and extensions for building scalable big‑data solutions.
AceroApache ArrowData Systems
0 likes · 20 min read

This article explains why new data systems are needed, introduces Apache Arrow’s columnar in‑memory format and its zero‑copy advantages, describes how to model data at read time, outlines the execution flow with Acero and SQL planning, and shares practical tips and extensions for building robust, dynamic‑schema data platforms.
AceroApache ArrowData Systems
0 likes · 20 min read

The article explains how Apache Arrow’s columnar, cross‑language in‑memory format enables high‑performance, interoperable data systems—replacing traditional row‑oriented databases—by supporting dynamic schemas, zero‑copy data exchange, efficient indexing, Acero‑based query execution, and Flight/ADBC connectivity, while offering practical guidance and highlighting challenges.
Apache ArrowData SystemsMemory Format
0 likes · 20 min read

This article explains why new data systems are needed, introduces Apache Arrow and its columnar in‑memory format, describes read‑time modeling and dynamic schema handling, and shows how Arrow can be used to build a complete data processing pipeline with indexing, SQL planning, and zero‑copy data exchange.
Apache ArrowData SystemsSQL
0 likes · 20 min read

The article explains how database expression evaluation—especially in WHERE and SELECT clauses—can be dramatically accelerated by replacing interpreted AST traversal with Just‑In‑Time compilation using Apache Gandiva, which leverages LLVM to generate SIMD‑optimized machine code for Arrow columnar data, and discusses extensions such as timestamp, array, higher‑order functions, and UDF support.
Apache ArrowApache GandivaExpression Evaluation
0 likes · 17 min read

This article explains how Just‑In‑Time (JIT) compilation, particularly via the Gandiva expression compiler built on LLVM and Apache Arrow, can dramatically accelerate database expression evaluation by transforming abstract syntax trees into native vectorized code, addressing traditional interpretation bottlenecks and improving CPU‑bound query performance.
Apache ArrowExpression EvaluationGandiva
0 likes · 17 min read

The article presents an interview with Wu Li, a senior R&D engineer at Shanghai Yanhuang Data, discussing how columnar storage, JIT compilation, and push‑mode execution are reshaping database performance in the era of big‑data analytics and evolving hardware constraints.
Apache ArrowGandivaJIT Compilation
0 likes · 10 min read

The article presents an interview with Wu Li, a research engineer at Shanghai Yanhuang Data, discussing how hardware limits have driven database evolution toward columnar storage, the adoption of Apache Arrow and Gandiva for SIMD‑enabled JIT compilation, and the shift from pull to push processing modes to improve OLAP performance.
Apache ArrowGandivaOLAP
0 likes · 10 min read

This announcement introduces Li Chenxi, a big‑data R&D engineer, and outlines his talk on leveraging Apache Arrow’s columnar in‑memory format to efficiently construct modern, read‑time modeling data systems, highlighting key features, ecosystem, and practical implementation benefits for the audience.
Apache ArrowColumnar MemoryData Systems
0 likes · 2 min read


phodalOct 31, 2022 · Frontend Development
The article examines the open‑source Perspective library—originating from J.P. Morgan’s FinOS—and its multi‑language architecture that combines C++/WASM, Rust, JavaScript, and Python to deliver high‑performance, framework‑free data visualizations for real‑time financial analytics, highlighting its use of Apache Arrow, Web Components, and Jupyter integration.
Apache ArrowC++Python
0 likes · 8 min read

The article examines three key design dimensions of modern AI platforms—choice of programming language, runtime environment isolation, and model deployment—highlighting how Python’s dominance, container‑based resource management, and efficient data sharing shape platform architecture and performance.
AI platformsApache ArrowKubernetes
0 likes · 13 min read

This article explains how SparkR’s performance compares to native Spark APIs, shows the slowdown caused by JVM‑R serialization, and demonstrates how enabling Apache Arrow’s vectorized execution in Spark 3.0 can accelerate SparkR operations by up to dozens of times.
Apache ArrowSparkRperformance
0 likes · 7 min read

Apache Arrow 1.0.0, the 18th major release, introduces binary‑stable columnar format changes, new metadata version V5, unsigned dictionary indices, a Feature enum, optional LZ4/ZStandard compression, expanded decimal bitWidth support, removal of validity bitmaps, and broader language bindings, enhancing big‑data analytics performance.
Apache Arrowcolumnar formatdata interoperability
0 likes · 3 min read
