Optimizing Apache Kylin for High‑Performance OLAP in Meituan's Sales System

Background : Since 2016 Meituan's in‑store dining team used Apache Kylin as an OLAP engine. Rapid business growth caused severe build‑time and query‑time inefficiencies.

Problem & Goal : The sales system “Qingtian” handles massive multi‑dimensional data, requiring precise deduplication and supporting thousands of concurrent queries. The aim is to improve data‑output efficiency and SLA from 90 % to 99.99 %.

Key Issues identified:

Difficulty locating root causes of inefficiency.

Build engine still based on MapReduce, not Spark.

Unreasonable resource usage and small‑file explosion.

Long build time (>2 h per day) for large fact tables.

SLA not meeting targets.

Optimization Approach :

1. Principle Understanding : Reviewed Kylin’s pre‑computation and By‑layer algorithm. The pre‑computation creates Cuboids for every dimension combination (2^N). By‑layer computes higher‑level Cuboids from lower‑level results, reducing redundant work.

2. Process Decomposition : Focused on five stages – engine selection, data reading, dictionary building, layered build, and file conversion – and refined each.

Engine Selection : Migrated the build engine from MapReduce to Spark (supported since Kylin 2.0). Spark improves build speed by 1‑3×.

Data Reading : Kylin reads Hive tables stored in HDFS. Merged small Hive files and tuned Map parameters to reduce task count.

Dictionary Building : Created dimension dictionaries to encode values and reduce HBase storage. Adjusted global dictionary dependencies for precise deduplication.

Layered Build : Adopted By‑layer algorithm exclusively under Spark. Each Cuboid layer becomes a Spark Job; each Job contains two Stages (read cache, write cache). Task parallelism is calculated as: CPU: 1*1000=1000; Memory: (6+1)*1000=7000GB Resource allocation formulas consider MapSize, cut‑mb, MaxPartition, MinPartition, and executor memory.

File Conversion : Converted built Cuboid files to HBase HFile via a MapReduce bulk‑load, aligning file count with the number of output partitions.

Implementation Roadmap : Conducted a pilot on the core sales transaction task, achieving a two‑way reduction in both resource consumption and execution time.

Results :

Overall resource usage dropped significantly across >20 Kylin jobs.

SLA reached 100 % in June 2020.

Outlook : Kylin upgraded to version 3.1 in July 2020, introducing Flink as the build engine, further improving efficiency.

Author : Yue Qing, Meituan in‑store dining R&D engineer since 2019.