Building a Data System with Apache Arrow: Design, Modeling, and Execution

The article begins by questioning the need for new data systems, citing the limitations of a "One Size Fits All" approach and the rapid emergence of new databases, which makes constructing specialized systems essential.

It then introduces Apache Arrow, describing it as a high‑performance, language‑agnostic columnar in‑memory format that enables zero‑copy data sharing across processes and languages.

Key concepts such as row‑wise vs. column‑wise storage, dynamic schema (read‑time modeling), and the challenges of handling heterogeneous log formats are discussed, with examples of how Arrow’s RecordBatch and schema flexibility address these issues.

The article outlines the architecture of a data system built on Arrow, covering execution flow (query parsing, logical and physical planning), storage using Parquet with Arrow compatibility, indexing (timestamp and inverted indexes), and the use of Acero as a push‑based execution engine.

It details extensions made to Arrow, including schemaless sink nodes for dynamic data, delayed output schema generation, and custom aggregation functions, as well as plans to support materialized views.

Data exchange mechanisms such as Arrow Flight, Flight SQL, and the upcoming ADBC are presented as ways to avoid costly row‑column conversions when communicating with clients.

Practical tips and pitfalls are shared, emphasizing the importance of contributing to the Arrow community, understanding the stability of its Core, Compute, and Acero layers, and considering the Rust implementation and DataFusion engine for richer SQL capabilities.

Finally, the article concludes that Arrow provides a solid foundation for modern data systems, enabling efficient storage, processing, and interoperability while reducing development effort.