Interview with Wu Li on Columnar Storage, JIT Compilation, and Push Mode in Modern Database Systems

Wu Li, a research engineer at Shanghai Yanhuang Data with a master’s degree from Shanghai Jiao Tong University, discusses the evolution of databases driven by hardware constraints, noting that while earlier bottlenecks were disk and network bandwidth, today CPU performance has become the primary limitation.

The conversation explains the shift from row‑oriented to column‑oriented storage, emphasizing that columnar formats store each column’s values contiguously, enabling faster aggregation, better compression, and SIMD‑based parallel computation. After evaluating formats such as Parquet and Avro, Yanhuang Data chose Apache Arrow for its in‑memory, language‑agnostic, and hardware‑aware design.

To accelerate expression evaluation in OLAP queries, the team adopted JIT (just‑in‑time) compilation. They leveraged the open‑source Gandiva library from Arrow for expression optimization, contributed enhancements back to the project, and addressed limitations by adding custom functions and improving registration mechanisms.

The article also contrasts pull‑mode (user‑driven data retrieval) with push‑mode (producer‑driven data delivery). By replacing pull‑mode operators with push‑mode equivalents, Yanhuang Data achieved noticeable query‑performance gains, especially for streaming and cache‑intensive workloads.

All these technical advances—columnar storage, JIT compilation, and push‑mode processing—serve a single product goal: delivering faster, reliable query results for large‑scale data analytics. The team stresses that end users care only about query speed and experience, not the underlying implementation details.

Overall, Yanhuang Data’s roadmap aligns with broader OLAP trends toward standardization and integrated data platforms, aiming to provide efficient heterogeneous data handling from ingestion through visualization.