Innovative Solutions for Reducing Result Inconsistency in Baidu Search System

Baidu's search system, a large-scale multi‑active distributed service, must deliver consistent results while handling massive query traffic. Inconsistent results degrade user experience, so accurately measuring and eliminating the sources of inconsistency is critical.

The team defines a "diff rate" metric by sampling a tiny fraction of incoming queries (the "user diff‑rate small traffic"). Each sampled query is duplicated into a pair (queryA, queryB) that are processed simultaneously. If the URL sequences of the two results differ, the query is marked as a "diff query." Over a period, diff rate = M / N, where N is the total sampled queries and M the number of diff queries. This metric is visualized as a diff‑rate curve.

In practice the diff‑rate curve is significantly above zero, often showing sudden spikes or prolonged high values. The root causes are hard to pinpoint due to the system's scale, complex query processing pipeline, and diverse intermediate data formats.

Conventional approaches—offline testing, full‑scale tracing/logging, or feature‑level dump analysis—are either too coarse, incur prohibitive performance/storage costs in production, or provide only correlation without causation.

To overcome these limitations, the authors propose an end‑to‑end solution built around **feature flattening experiments** performed directly in production. The key ideas are:

Quantify the contribution of each service or feature to the overall diff rate by flattening (making results identical) that component and observing the change in diff rate.

Use a tiny, controlled traffic slice (both real and fake) to run experiments without affecting user experience.

Introduce a dynamic debugging mechanism that collects detailed debug data (similar to Dapper) from fake traffic.

Coordinate many experiments through a top‑down, hierarchical decomposition of the system.

Automate the entire loop with an "automatic inspection flywheel" that decides experiment configurations, runs them, and generates contribution reports without human intervention.

Implementation highlights include:

Traffic coloring: each query carries a marker indicating whether it should interact with the flattening server at a specific processing stage.

Dynamic role assignment: the first arriving packet becomes the publisher, the later one the subscriber, enabling symmetric handling.

Data flattening processing: the flattening server stores both packets, returns the publisher unchanged, and modifies the subscriber according to predefined rules to achieve result equality.

For single flatten experiments, a unified feature‑location description rule simplifies configuration, and multiple flattening strategies are abstracted for different data shapes.

To achieve a zero‑explosion‑radius (no impact on real users) while preserving experiment reliability, the authors decouple real diff‑rate traffic from fake traffic. Real traffic provides the true diff rate, while a separate fake‑baseline traffic establishes a reference. Experiments run on fake‑experiment traffic only when its baseline matches the real diff rate, ensuring conclusions are trustworthy.

Dynamic debug leverages fake traffic to collect exhaustive debug data, enabling deep root‑cause analysis beyond mere contribution measurement.

Multi‑experiment orchestration follows a top‑down approach: the system is white‑boxed, layers are flattened iteratively, and the most contributing sub‑components are further decomposed until contributions fall below a threshold. Identified high‑impact features are then targeted.

Traffic space reuse partitions the overall experiment traffic into static buckets (determined by query ID modulo N). Each bucket maintains its own diff‑rate curve and can be independently flattened, with service‑mesh routing handling bucket selection and experiment configuration.

The automatic inspection flywheel consists of two parts: (1) automated decision making that dumps request/response packets to the flattening server, performs flattening, and stores results; (2) automated report generation that periodically extracts bucket diff‑rate data, computes feature contributions, and produces actionable reports. This pipeline eliminates manual effort.

Results show that the solution captured all features with substantive contributions to the diff rate, dramatically reduced result volatility, and provided precise guidance for system optimization. The approach also offers a reusable framework for other large‑scale distributed systems facing consistency challenges.

In summary, the paper presents a comprehensive, production‑grade methodology for quantifying and eliminating result inconsistency in a distributed search engine, combining data flattening, fake traffic, dynamic debugging, coordinated experiments, and full automation.