Operational Topic Data Productization: Architecture, Modeling, and Service Layers at Meituan-Dianping

Meituan-Dianping, the world’s largest local life service platform, faces massive POI data and a huge number of active users. Under such massive data volume, locating the required operational data, providing consistent and readable data quickly, and supporting diverse business analysis become extremely challenging.

Key challenges identified:

High threshold for data extraction: difficulty finding suitable data, complex metrics, and high skill requirements increase labor costs.

Time‑consuming data processing: lack of offline warehouse models and pre‑computation support leads to slow ad‑hoc queries.

Data inconsistency: different channels use different metric definitions, lacking unified management.

Unfriendly data feedback: insufficient visualization makes trend analysis and multi‑dimensional comparison hard.

To address these problems, the team proposed productizing the operational topic data and designed a comprehensive solution.

Solution Overview

The solution builds a public underlying data layer for the domestic tourism operation side, integrates data from various promotion systems, categorizes activities by theme, aggregates events, and provides a unified data export. Multi‑dimensional pre‑computation engines are used to pre‑aggregate facts, reducing computation cost and improving data consumption efficiency. Finally, a unified data service middle‑platform and web applications are built, offering rich visualizations and diverse analysis capabilities.

Data Warehouse Layer

The core of the product is the data warehouse model, which directly influences product feasibility, data consistency, and usability. Four common modeling methods are introduced:

3NF model : Inmon’s normalized model focusing on entity‑relationship representation and data consistency governance.

Dimensional model : Kimball’s star/snowflake schema for fast analytical queries and pre‑aggregation.

DataVault (DV) model : Linstedt’s hybrid model balancing flexibility, scalability, and auditability (Hub, Link, Satellite).

Anchor model : Rönnbäck’s further normalized extension of DataVault, using narrow tables (Anchors, Attributes, Ties, Knots).

Based on the business needs, a hybrid 3NF + Dimensional modeling approach is adopted to construct the operational topic data system.

Data Construction Process

1. Data Standardization : Establish a metric dictionary and service‑layer rule engine, reach consensus on metric definitions, and unify metric granularity (e.g., GMV definitions).

2. Warehouse Architecture : The warehouse is divided into four layers:

ODS layer : Consumes binlog and click‑log from distributed message queues, cleanses data, restores business tables, and syncs to Hive.

BAS layer : Applies 3NF modeling to abstract business concepts, reduce redundancy, and ensure data consistency.

FACT layer : Uses dimensional modeling to integrate coupon, cash‑voucher, and promotion events, providing pre‑processed dimensions for fast ad‑hoc queries.

TOPIC layer : Optional layer for personalized topic data serving specific business groups.

3. Pre‑computation Layer : Built on Kylin, creates OLAP cubes (e.g., transaction cube with date, user, payment type, city) to lower data extraction thresholds and accelerate multi‑dimensional analysis.

Middle‑Platform Service Layer

To decouple front‑end applications from data processing, a plug‑in service layer is introduced, consisting of:

Configuration Center : Centralized management of resources (databases, services, caches) and environment‑specific settings.

Metric Dictionary : Unified naming and definition of metrics, enabling consistent calculations across teams.

Rule Engine : Extracts business rules from hard‑coded logic, supports data preparation rules (e.g., conditional red‑packet distribution) and calculation rules that generate derived metrics.

Compute Engine : Core module that shards and buckets data, processes multi‑dimensional queries in parallel, and isolates computation from business logic.

Memo (Snapshot) Store : Periodic snapshots of computed data for trend analysis and historical comparison.

Data Visualization

Using open‑source ECharts, the product provides interactive visualizations such as trend comparison, dimensional reduction, metric comparison, and multi‑dimensional query interfaces, turning raw numbers into actionable insights.

Benefits

Unified data standards ensure consistent metric definitions across teams.

High scalability supports the entire internal operations team and diverse business needs.

Unified data and middle‑platform services enable flexible configuration and rapid iteration.

Reduced development, storage, and R&D costs through layered modeling and domain‑driven data partitioning.

Rich visual analytics empower business users with multi‑dimensional, drill‑down, and comparative analyses.

Recruitment

The team is continuously hiring data warehouse and data development engineers. Interested candidates can send resumes to yangguang09#meituan.com.