Experience and Optimization Strategies for Apache Kylin in Real-Time OLAP

The author, a data development engineer at Qunar.com, explains the need for real‑time, interactive multidimensional analysis on PB‑level data, highlighting limitations of pre‑computation with Hive and the necessity for a scalable solution.

Apache Kylin is introduced as an open‑source distributed analytics engine built on Hadoop that provides SQL query interfaces and pre‑computed cubes, enabling fast queries on massive datasets.

The implementation experience covers business model abstraction, where raw data is cleaned, transformed, and loaded into fact tables that follow a star schema, then used to build wide tables for cube construction.

Cube dimension explosion is discussed, illustrating how the number of possible cuboids grows exponentially (2^N) and leads to storage and performance issues, prompting the need for dimension reduction strategies.

Three primary dimension‑reduction methods are presented: pruning unnecessary dimensions, splitting cubes into smaller sub‑cubes, and applying hierarchy, mandatory, and joint dimension optimizations to limit cuboid combinations.

Additional optimization topics include handling Count Distinct metrics, where Apache Kylin offers both approximate HyperLogLog and exact bitmap‑based calculations, each with trade‑offs between accuracy, storage, and query performance.

The article concludes with practical tips and lessons learned from deploying Kylin in production, emphasizing the balance between space‑time trade‑offs and the importance of informed cube design.