Meituan-Dianping Real-Time Data Warehouse Platform Built on Apache Flink: Architecture, Practices, and Future Directions

This article presents Meituan-Dianping’s real‑time data warehouse (实时数仓) built on Apache Flink, beginning with an abstract that highlights the importance of data warehouses for data intelligence and the challenges of large‑scale data applications.

The first major section outlines the evolution of Meituan-Dianping’s real‑time computing platform: starting with Storm in 2016, adding Spark Streaming in early 2017, and adopting Flink at the end of 2017, emphasizing improvements in safety, stability, and usability.

The platform architecture is described in five layers: collection (Binlog, logs, IoT data into Kafka), storage (Kafka, HDFS, HBase), engine (Storm, Flink with framework wrappers), platform (managing data, tasks, resources), and application (real‑time warehouse, ML, data sync, event‑driven apps).

Job management includes configuration, publishing (version control, compile/release/rollback), and status monitoring (runtime state, custom metrics, alerts, logs). Resource management provides multi‑tenant isolation and resource delivery/deployment capabilities.

Business warehouse practices are illustrated with three examples: traffic warehouse (log collection, channel splitting, real‑time analysis), ad real‑time effect verification (join logs via requestID, store in Druid for CTR analysis), and instant delivery (feature extraction for delivery time prediction using Storm).

The article compares traditional, real‑time, and near‑real‑time warehouse models, detailing layer structures (ODS, DWD, DWS, application) and storage choices (Kafka for fact data, KV stores for dimensions, Flink for on‑the‑fly queries).

A side‑by‑side comparison of near‑real‑time (OLAP‑based) and real‑time (stream‑processing) warehouses discusses scheduling overhead, flexibility, tolerance to late data, scalability, and suitable scenarios.

Reasons for choosing Flink are presented: mature state management, rich table APIs (Stream, Table, SQL), comprehensive ecosystem support, and unified batch‑stream processing.

The platform’s components are detailed: message expression (unified protocol, binlog sharding), compute expression (extended DDL for metadata, unified storage access), UDF platform (security auditing, quality testing, reuse/versioning), and a Web IDE that enables SQL‑based development with version control.

Future development focuses on resource auto‑tuning for a cluster of thousands of nodes, dynamic scaling based on traffic peaks, and integrating real‑time and offline workloads with fine‑grained isolation.

Finally, the roadmap for upgrading real‑time warehouse construction is outlined, emphasizing automated modeling, unified technical expression, and the need to advance beyond the current implementation.