How Meituan Waimai Scaled Feature Engineering for Billions of Requests

Background

Meituan Waimai operates recommendation, advertising, and search business lines for billions of users. Rapid algorithm iteration requires a unified feature platform that can generate, compute, and serve features at massive scale while keeping user relevance, merchant exposure, and platform revenue balanced.

Feature Platform Architecture

Three‑stage Pipeline

Training Sample Generation (offline) : Configurable pipeline that produces labeled training data for model validation.

Feature Production (near‑line) : High‑throughput processing, scheduling, storage and synchronization of billions of feature records.

Feature Retrieval & Computation (online) : Low‑latency, high‑availability fetching and transformation of features for model inference.

Feature Production Core Mechanisms

Feature Semantics and Merging

Each feature is defined by four fields: source Hive table, dimension, filter condition, and aggregation condition. The platform parses these configurations into SQL abstract syntax trees, applies equivalence rules (e.g., commutativity), and merges identical queries into a single SQL statement. The merged result is written to a shared feature table, eliminating duplicate data pulls.

Multi‑Task Scheduling

The platform orchestrates three task types:

Semantic Tasks : Pull raw data from Hive, apply the semantic definition, and write to the shared table.

Aggregation Tasks : Group features per tenant (business line), generate full snapshots and incremental updates.

Synchronization Tasks : Sync incremental/full data to KV stores with rate‑limiting, health checks and version handling.

A scheduler monitors upstream Hive readiness, prioritizes high‑impact semantics, and selects the least‑loaded execution queue to maximize throughput.

Feature Storage

Features are serialized with Protobuf for compactness and speed. Instead of static .proto files, the platform uses a dynamic serialization layer that reads feature metadata at runtime, allowing new features to be added without rebuilding JARs. Both single‑version (overwrite) and multi‑version storage are supported; the platform adopts multi‑version to enable instant rollback during anomalies.

Online Feature Retrieval & Computation

Online services fetch grouped feature data from KV stores; intelligent grouping reduces the amount of data read per request and stabilizes network bandwidth. The platform introduces Model Feature Description Language (MFDL) , a DSL that separately describes feature calculation and transformation, enabling hot‑updates of feature logic without redeploying models.

MFDL supports primitive types (String, Long, Double), Map and List (embedding) structures, arithmetic and logical operators, and custom function operators. The DSL is compiled by a JavaCC‑based engine that performs vectorization and parallel execution.

Training Sample Construction

Configuration‑Driven Pipeline

The pipeline follows a template‑plus‑configuration model with three stages: Input, Transform, and Output. Key components include:

Label preprocessing via Hive/Spark SQL with built‑in UDFs (e.g., URL decode, MD5).

Experimental feature extraction (feature补录) for new features not present in the consistency sample.

Feature‑sample join on identifiers (e.g., poi_id).

Feature matrix generation using MFDL configurations.

Format conversion to CSV, TFRecord, etc.

Statistical analysis (mean, variance, quantiles, null rate).

Data write‑out to Hive/HDFS.

Each component can be toggled via configuration, allowing the pipeline to start from any intermediate node.

Consistency Guarantees

Inconsistencies between offline training and online prediction arise from divergent configs, update timing, and operator definitions. The platform resolves these by:

Using a single MFDL file for both online and offline feature computation.

Generating training samples from real‑time online feature snapshots.

Maintaining a shared operator library for both environments.

Feature Discovery & Business Empowerment

A discovery system scores features on reuse degree, annotation completeness, and data‑quality metrics. Top‑ranked features are recommended to users, and a dynamic grouping mechanism ensures efficient storage and retrieval.

Key Technical Highlights

Semantic Extraction & Merging : Automatic SQL AST generation and equivalence detection to collapse duplicate feature pulls.

Multi‑Task Scheduler : Prioritizes semantics, monitors upstream readiness, and balances load across queues.

Dynamic Protobuf Serialization : Runtime metadata lookup replaces static .proto files, enabling zero‑downtime feature addition.

Multi‑Version KV Storage : Versioned feature records allow instant rollback without full data rewrite.

MFDL DSL : Expressive language for feature calculation (supports primitives, Map, List, arithmetic, logical ops, custom functions) compiled to vectorized, parallel Java code.

Intelligent Feature Grouping : Business and public groups adaptively reorganized based on usage patterns to minimize KV read size and protect cluster stability.

Consistent Sample Generation : Unified MFDL, real‑time snapshot ingestion, and shared operator library guarantee offline‑online parity.

Future Directions

Ongoing work focuses on further optimizing the end‑to‑end algorithm loop, extending multi‑version storage strategies, and automating feature‑value contribution analysis for deep‑learning models.