Real-Time Data Warehouse Practices: Case Studies from Meituan, NetEase, Zhihu, and OPPO

The concept of data warehouses originated decades ago, with traditional offline warehouses evolving into offline warehouses built on Hive/HDFS and, more recently, into real‑time warehouses powered by streaming frameworks such as Flink, Storm, and Spark Streaming.

Four representative real‑time warehouse implementations are highlighted:

Meituan : A Flink‑based real‑time platform that collects data from binlog, service logs, and IoT devices into Kafka, stores state in HDFS and HBase, and provides job configuration, publishing, and monitoring functions across collection, storage, engine, platform, and application layers.

NetEase (Yanxuan) : An architecture that follows the classic ODS‑DWD‑DWS‑DM layering, using Kafka for ODS and DWD, Redis for dimension data, HBase for high‑concurrency queries, and MySQL/Greenplum for aggregated metrics, with Flink handling real‑time ETL and aggregation.

Zhihu : Evolution from version 1.0 (Spark Streaming, Druid for metrics) to version 2.0 (Flink Streaming, SQL‑based processing). The upgrade addresses Kafka traffic overload, Druid stability, and lack of data isolation, and introduces Streaming SQL, metadata management, and automated result verification.

OPPO : A smooth migration from offline to real‑time warehouses, using Kafka as the central message bus, Flink SQL for cleaning and aggregation, and a layered design (ODS → DWD → ADS) that ultimately feeds downstream systems such as Elasticsearch, MySQL, and Hive.

Across these cases, common design principles emerge: a four‑layer data model (ODS, DWD, DWS, ADS/DM), the preference for Kafka as the real‑time storage backbone, and Flink as the preferred streaming engine due to low latency, exactly‑once semantics, and rich SQL support.

The article concludes that building a real‑time data warehouse requires careful layer planning, appropriate technology selection (Flink, Kafka, HBase, Redis, MySQL, Hive), and continuous iteration to balance latency, scalability, and operational stability.