Building a Quality Model for Meituan's Recommendation System

1 Introduction

Meituan's integrated business recommendation platform (referred to as "to‑store") serves dozens of local‑life categories and relies on a data‑flywheel: data warehouse → algorithm processing → service layer → client feedback. The quality of this pipeline directly impacts matching effectiveness, prompting the need for a fine‑grained quality model.

2 Current Situation Analysis

Traditional availability metrics (e.g., N‑9) focus on service uptime and miss many defects relevant to effect‑oriented recommendation systems. They are insensitive to data‑quality issues, cannot capture business‑specific variations, and overlook defects such as model‑update latency or crawler traffic that distort core KPIs like click‑through and transaction conversion.

3 Construction Idea

3.1 Quality in Business Context

Following ISO definitions, quality reflects the ability of a system to meet explicit or implicit needs. In recommendation, the needs are product‑level (user‑facing) and algorithm‑level (model performance). The model shifts the focus from a binary "service available / unavailable" to a spectrum of "good / bad" outcomes.

3.2 Defect Consideration and Selection

Defects are derived from ISO/IEC 25010 features. For recommendation, the most relevant features are accuracy, appropriateness, and security (e.g., crawler‑induced noise). Figure 3 in the original article lists the selected quality characteristics.

System layer – exceptions such as recall timeout, empty results.

Data layer – anomalies in supply quantity or label distribution, evaluated via data‑lineage.

Algorithm layer – feature, model, or strategy issues like delayed model updates.

Business layer – violations of business appropriateness or safety, e.g., severe bad cases.

3.3 Metric and Calculation Selection

Availability can be expressed as N‑9 or as a function of mean time between failures. The article adopts a request‑level metric: the proportion of successful (non‑defective) requests. Figure 5 illustrates aggregating quality across multiple dimensions using request‑level functions.

4 Calculation Method

4.1 Formula

The quality score is computed as the ratio of non‑defective requests to total requests, with defects classified into the four layers described above. Figure 6 shows the concrete calculation flow.

4.2 Business Generalization

Because the platform serves many low‑traffic business lines, the model aggregates metrics at industry or project level to reduce volatility, enabling minute‑level monitoring for small‑traffic slots.

4.3 Indicator System

The top‑level quality score is derived from request‑level success rate. First‑level input metrics include recall defect rate and ranking defect rate; second‑level metrics further break down these rates (e.g., empty recall, timeout). Business‑dimensional aggregation yields vertical, horizontal, and temporal quality scores.

4.4 Data‑Lineage Expansion

Beyond request‑level statistics, the model extends to full data‑lineage, linking tables → services → client traffic. Six business metadata types (snapshot, plan, component, index, model, feature) are used to build a graph database; anomalies propagate along the lineage to affect the quality score.

5 Metric Operations

5.1 System Implementation

Data collection relies on instrumentation at each stage (parameter input, pre‑recall processing, recall, post‑recall, coarse ranking, fine ranking, re‑ranking). Kafka streams raw logs; near‑real‑time Elasticsearch indexes the last four days for fast queries, while older logs are archived to Hive. Flink parses the logs, performs diagnostics, computes the quality score, and pushes alerts. Test environments route logs to MySQL for debugging.

5.2 Alert Follow‑up

To avoid alert storms, alerts are routed to owners of specific traffic slots, and follow‑up status is recorded. This enables timely awareness and post‑mortem analysis.

5.3 Governance Effects

By distinguishing empty‑result causes (supply vs. quality), 98 % of empty results are excluded from the score, reducing daily empty‑result alerts from 40 to 5. Governance actions (data standardization, service isolation, change‑pipeline checks) keep the empty‑result detection rate above 60 % and achieve a core‑traffic empty‑result rate of 0.01 % (99.99 % successful requests).

5.4 Asset Accumulation

The quality model also serves as a data‑asset repository: collected data become assets that are flowable, measurable, controllable, and value‑adding. Analysis reveals that 80 % of online incidents stem from releases, and 80 % of those are data‑release issues, guiding future data‑release governance.

6 Future Planning

Enrich the quality model with more indicators and broader coverage.

Develop flexible degradation strategies and quantify their impact on the quality score.

Improve data‑lineage accuracy, coverage, and timeliness for faster impact assessment.

// Parameter diagnosis primitive example (Aviator expression)
global:check=aviator[cityId != nil && include(string.split('1,2,3,4,5,6,7,8,9,10,16,17',','), str(cityId))]

// Recall exception diagnosis
global:recallException=param[${recall#exception#}]
global:check=aviator[recallException != nil && recallException != '']