Building a Complete Data System with Apache Arrow: Architecture, Dynamic Schema Modeling, and Practical Tips

Why Build a New Data System – Modern workloads require specialized systems because a single "One Size Fits All" solution cannot handle diverse use cases such as OLTP, OLAP, streaming, and cloud‑native workloads. The rapid emergence of new databases (124 since 2020) illustrates the need for continual innovation.

Read‑Time Modeling – Instead of pre‑defining schemas (write‑time modeling), logs from heterogeneous sources are stored raw and schemas are inferred at query time, enabling flexible handling of varied log formats.

Memory Data Formats – Arrow provides a high‑performance columnar in‑memory representation that works across languages (C++, Python, Rust, etc.). Row‑oriented storage suits transactional workloads, while columnar storage reduces I/O for analytical queries and improves cache locality.

Advantages of Apache Arrow – Zero‑copy inter‑process communication, standardized handling of nulls, timestamps, and complex types, broad language support, and integration with projects such as PyTorch, Spark, ClickHouse, and DuckDB.

System Architecture – The data pipeline includes event‑based storage (Parquet), dynamic RecordBatch schemas, timestamp and inverted indexes, and a custom SQL parsing layer (ANTLR). Logical plans are generated, optimized, and translated into physical plans executed by Arrow’s Acero engine.

Extensions and Tips – Added schemaless SinkNode for dynamic schemas, delayed output schema generation, and extensions to aggregation functions. Recommendations include contributing to Arrow, using the Rust implementation with DataFusion for a full SQL engine, and leveraging Arrow Flight/Flight‑SQL for efficient columnar data exchange.

Practical Lessons – Arrow’s core layer is stable, compute layer may have bugs, and Acero is still experimental. Complex types (Union, List, JSON) need extra work. Community activity is high, but careful testing on large‑scale data is advised.