Applying Apache Kylin for Large‑Scale OLAP at Meituan: Architecture, Challenges, and Performance Evaluation

1 Meituan Data Scenario Characteristics

Meituan’s business lines generate billions of rows of data in Hadoop for OLAP analysis, serving thousands of analysts and city BD users with high demands on scalability, stability, accuracy, and latency.

Data scale and model: Fact tables range from 100 million to 1 billion rows, with dimension tables of tens of millions and high‑cardinality dimensions. Many models are snowflake or "constellation" rather than simple star schemas, making classic modeling difficult.

Dimension volatility: Dimensions often change (e.g., organizational structures), requiring frequent re‑processing of historical data.

Data back‑tracking: Errors in upstream data necessitate re‑runs, which differ from traditional static models.

Metric constraints: Typically fewer than 50 metrics, but many are expression‑based, which Kylin’s native aggregation cannot handle directly. Precise deduplication is required, so approximate algorithms like HyperLogLog are unsuitable.

Query requirements: Interactive analysis with second‑level response times for tens of thousands of concurrent queries.

2 Solution for Integrating Apache Kylin

Wide‑table approach: Non‑star models are flattened into a wide table via pre‑processing, allowing Kylin to aggregate efficiently and handling high‑cardinality dimensions.

Expression metric handling: Expressions are materialized as separate columns (via Hive views or physical tables) before loading into Kylin.

Precise distinct counting: Implemented using Bitmap for integer types; later versions (1.5.3) added full‑type support.

Deployment architecture: Kylin servers are deployed in a split‑mode: a production server (kylin01) handles query serving and cube building, while a pre‑release server (kylin02) is used for development and review before promotion.

3 Mainstream OLAP System Comparison

Benchmarks using the SSB dataset (star‑schema) compared Presto, Kylin 1.3, Kylin 1.5, and Druid across three data scales (10 M, 1 B, 10 B rows) and five query patterns. Results show:

Kylin 1.5 and Druid outperform Presto on OLAP‑specific workloads.

Kylin 1.5 benefits from parallel HBase scans, scaling better than 1.3 on larger datasets.

Druid excels in low‑latency queries due to in‑memory storage, but lacks precise distinct counting.

Beyond performance, the evaluation considered functionality completeness, ease of use, data preparation cost, and query flexibility.

4 Advantages of Kylin

Stable performance: Leveraging mature Hadoop components (Hive, HBase) yields >99.99% availability and sub‑second latency for 95% of queries.

Exact data accuracy: Only Kylin currently provides fully precise distinct counts at Meituan’s scale.

Operational simplicity: Minimal additional infrastructure, web UI for cube management, and standard SQL interface reduce deployment and maintenance overhead.

Active community: Strong contributions from Chinese engineers, including Meituan, and a vibrant Apache Kylin community.

5 Future Work

Planned improvements include full‑type bitmap distinct counting, better resource isolation for multi‑tenant clusters, handling large result sets with pagination, and exploring streaming and detailed data features introduced after Kylin 1.5.

6 Q&A Highlights

Typical build time for 200 M rows with ~15 dimensions is under two hours, producing 500‑600 GB of cube data; after compression the size drops to a few gigabytes. Kerberos tickets need periodic refresh but do not require service restarts.