Exploring OLAP Engine with Apache Kylin: Architecture, Theory, and Applications in Qunar's Big Data Platform

Qunar's data team faced severe performance bottlenecks when using a Saiku+Mondrian engine backed by MySQL, as query response times grew to minutes with tens of millions of rows, prompting a need for a more scalable OLAP solution.

The new requirements emphasized visual drag‑and‑drop dimension composition, custom dimension queries, multi‑dimensional aggregation, sub‑second response, and support for petabyte‑scale data volumes.

Apache Kylin, an open‑source distributed analytics engine built on Hadoop, was introduced to meet these needs. Kylin provides ANSI‑SQL query interfaces, multi‑dimensional analysis, and sub‑second interactive queries over TB‑to‑PB datasets by pre‑computing cubes and storing results in HBase.

Kylin's architecture leverages Hive for source data extraction, MapReduce for cube building, and HBase for cube storage, exposing REST, JDBC, and ODBC APIs for downstream tools like Tableau and Excel.

The core theoretical principle of Kylin is "space for time": it pre‑computes all possible aggregations into cubes, turning complex joins and aggregations into fast look‑ups. Cubes are built layer‑by‑layer using a Layer Cubing algorithm, where each layer is a separate MapReduce job.

During query execution, Kylin rewrites SQL statements to target the appropriate cuboid, retrieving pre‑computed measures from HBase, which enables high concurrency and millisecond‑level latency.

In Qunar's airline ticket data scenario, Kylin powers a solution with six international ticket data cubes, 34 custom dimension tables, and a development cycle of 1–2 days per cube, delivering query latencies under 200 ms for tens of millions of rows.

The deployment also uncovered several pitfalls—such as dimension cardinality explosion, storage overhead, and cluster resource contention—and led to optimization practices including cube design guidelines, parameter tuning, and automated incremental data loading.