Interview with Wu Li on Database Evolution: Columnar Storage, JIT Compilation, and Push Mode

The interview begins by describing how the digitalization of enterprises is driven by database evolution, which historically moved from disk‑ and network‑bound bottlenecks to CPU‑bound constraints, prompting the adoption of distributed architectures and new storage models.

Wu Li explains that early columnar storage was a key breakthrough, allowing data in the same column to be stored contiguously, which simplifies aggregation, improves compression, and enables SIMD parallelism for faster calculations.

After evaluating formats such as Parquet and Avro, Yanhuang Data chose Apache Arrow for its in‑memory, language‑agnostic, hierarchical design, and integrated it with the Gandiva expression‑optimization library to accelerate projection and filtering operations.

Because Gandiva lacked some custom function support, Yanhuang contributed patches back to the open‑source project, enhancing its capabilities for their use cases.

The team also adopted JIT (just‑in‑time) compilation to compile expression DAGs at runtime, reducing linking overhead and improving execution speed, especially for complex SQL queries.

Finally, Yanhuang transitioned from a traditional pull‑mode data consumption model to a push‑mode architecture, rewriting execution operators to proactively process and deliver data, which yielded noticeable query‑performance gains across the platform.

Throughout, the focus remains on delivering a consistent, high‑performance user experience for large‑scale OLAP workloads by aligning hardware trends, open‑source tools, and architectural choices.