Apache Kylin at AutoHome: Development History, Architecture, Optimizations, and Future Plans

AutoHome adopts Apache Kylin as its core OLAP engine, serving multiple business lines and commercial data products for traffic, leads, user behavior, and recommendation effectiveness analysis, with over 500 cubes storing around 300 TB and handling queries with 95th‑percentile response times under 2 seconds.

The article first introduces Kylin’s background, architecture, and pre‑computation principles, explaining how data sources (Hive, Kafka, relational databases) feed into a REST‑based query layer, which routes SQL queries to pre‑built cubes using Apache Calcite for parsing and planning, and finally stores results in HBase.

It then describes the current usage status, highlighting the scale of cubes (500+, up to 31 dimensions per cube, trillions of rows) and the storage of millions of HBase regions.

The development history from 2016 to 2019 is outlined, showing progressive adoption, version upgrades, increased cube count, multi‑cluster deployment, and integration with internal BI tools.

Practical applications in commercial data products are discussed, including business scenarios, technology selection (Kylin vs. Druid vs. Elasticsearch), the strategic data product "CheZhiYun," development workflow, and common optimization techniques such as cuboid pruning, dictionary encoding, and precise deduplication.

Several optimization cases are presented: limiting maximum dimension combinations to reduce cuboid count, segment filtering to avoid loading unnecessary dictionaries, handling mixed precise and imprecise deduplication via hybrid cubes, disabling cubes during incremental builds, and improving scheduler performance by skipping already successful jobs.

Additional operational improvements include monitoring Kylin metrics with Prometheus/Grafana, automated health checks and restarts, HBase master‑slave replication for high availability, and the KylinSide tool for cluster statistics and management.

The article also shares experience with cluster upgrades and migrations, detailing background challenges, overall solution (parallel new‑cluster deployment, metadata sync, segment rebuilding, SQL replay), and architecture using KylinSide for automation.

Future plans focus on real‑time OLAP capabilities and cloud‑native deployment, aiming to integrate streaming receivers at the cube level and simplify real‑time multi‑dimensional modeling.