Understanding Search Experiments: AB Testing, Experiment Types, and Common Issues

The article introduces search experiments from a data‑product perspective, beginning with an overview of AB (A/B) testing, its purpose of risk reduction and quantitative evaluation, and the basic principle of randomly allocating traffic to control and treatment groups.

It explains the limitations of early single‑layer experiment platforms, such as traffic starvation, and describes the modern multi‑layer “layer+domain” architecture inspired by Google’s Overlapping Experiment Infrastructure, which enables orthogonal traffic allocation across layers.

The piece then defines key terminology used in search scenarios (query, intent judgment, result page, QV, card impact, strategy impact, card position) and outlines four main experiment types supported by the platform: ordinary AB, vocabulary (word‑list) experiments, diff‑AB, and interleaving, detailing the appropriate use cases for each.

Real‑world case studies illustrate how ordinary AB experiments are applied to result‑page redesigns, vocabulary experiments to card‑style changes, diff‑AB for strategy control, and interleaving for algorithm ranking, including descriptions of balanced and team‑draft interleaving methods.

The article concludes with a FAQ section addressing common problems such as ineffective experiment cards, traffic imbalance, low impact, experiment interference, negative metrics, slow data reporting, and system stability, offering practical troubleshooting advice.