Building a Data System with Apache Arrow: Design, Implementation, and Practical Tips

To address performance and interoperability challenges in large‑scale data processing, the article first discusses why a new data system is required, highlighting the limitations of a "One Size Fits All" approach and the rapid emergence of new databases in the current "golden age" of data systems.

It then introduces the concept of read‑time modeling, which stores raw logs without predefined schemas and extracts fields on demand, enabling flexible handling of heterogeneous log formats and dynamic schema evolution.

The core of the solution is Apache Arrow, a high‑performance, language‑agnostic columnar in‑memory format that provides zero‑copy data exchange across systems. Arrow’s design supports both row‑wise and column‑wise storage, making it suitable for transactional and analytical workloads, and it is widely adopted by projects such as PySpark, ClickHouse, and DuckDB.

The article outlines the architecture of a data system built on Arrow, covering memory data formats, storage using Parquet with dynamic schema extensions, index structures, and the execution pipeline: query parsing (using ANTLR), logical planning, optimization, and physical planning with the Acero engine. It also describes how to extend Acero with custom execution nodes and support dynamic schemas.

Practical considerations include using Arrow Flight/Flight SQL for efficient columnar data transfer, leveraging the emerging Arrow ADBC interface, and employing DataFusion for full SQL support and query optimization.

Finally, the article shares tips and pitfalls learned from the implementation, such as the importance of contributing back to the Arrow community, handling complex types, and the need for further validation at scale, while recommending the Rust implementation and DataFusion for modern, secure data processing workloads.