Interview with Wu Li on Columnar Storage, JIT Compilation, and Push Mode in Database Development

The interview begins with an overview of enterprise digitalization driven by database evolution, noting that hardware bottlenecks have shifted from disk and network bandwidth to CPU performance, making efficient data processing increasingly critical.

Wu Li explains the early adoption of columnar storage, contrasting it with row‑based storage: columnar layouts keep values of the same column together, enabling faster aggregation, better compression, and SIMD‑friendly parallelism, which significantly speeds up OLAP queries.

After evaluating formats such as Parquet and Avro, the team selected Apache Arrow for its in‑memory, language‑agnostic, hierarchical data model, and integrated it with the Gandiva expression‑optimization library to accelerate projection and filtering.

Recognizing limitations in Gandiva, Yanhuang Data contributed patches upstream and added custom functions, leveraging open‑source collaboration to improve the toolset.

The discussion then moves to JIT (just‑in‑time) compilation, describing how runtime code generation reduces interpretation overhead and improves expression evaluation, especially when combined with SIMD instructions, while noting the need for careful CPU‑architecture compatibility.

Finally, Wu Li describes the transition from pull‑mode (user‑driven data fetching) to push‑mode (producer‑driven data delivery), highlighting the engineering effort required to replace execution operators and the resulting comprehensive query‑performance gains across the product.

Throughout, the interview emphasizes that these technical advances—columnar storage, JIT compilation, and push‑mode execution—are pursued to deliver faster, more reliable query experiences for end users in large‑scale data environments.