Unlocking Randomized Experiments: Advanced Techniques to Boost Test Power

This comprehensive guide explores the fundamentals of randomized controlled experiments, discusses classic RCT designs and their limitations, and presents advanced methods such as CUPED variance reduction, stratified, paired, and covariate‑adaptive randomization, as well as spill‑over modeling and random saturation designs to improve experimental power and reliability.

A/B testingCUPEDRandomized Controlled Experiments

0 likes · 59 min read

Unlocking Randomized Experiments: Advanced Techniques to Boost Test Power

Didi Tech

Apr 10, 2025 · Product Management

AA Testing and Rerandomization Techniques for Reliable AB Experiments

The article outlines how AA testing and rerandomization can detect and correct non‑uniform traffic splits in short‑term AB experiments, detailing three solutions—AA tests, seed‑based rerandomization, and retrospective AA analysis—along with theoretical guarantees, empirical error‑rate reductions, and remaining challenges for long‑term or clustered designs.

AA testingAB testingCUPED

0 likes · 17 min read

AA Testing and Rerandomization Techniques for Reliable AB Experiments

DataFunSummit

Aug 18, 2024 · Artificial Intelligence

Challenges and Solutions in Recommendation AB Testing on Xiaohongshu's Experiment Platform

The article examines the key challenges of recommendation AB testing at Xiaohongshu—including change stability, single‑experiment precision, and multi‑strategy packaging—and presents a series of engineering and statistical solutions such as SDK‑based AB architecture, virtual PreAA experiments, CUPED/DID adjustments, and reverse experiments to improve reliability and metric impact.

AB testingCUPEDExperiment Platform

0 likes · 15 min read

Challenges and Solutions in Recommendation AB Testing on Xiaohongshu's Experiment Platform

Huolala Tech

Dec 29, 2023 · Fundamentals

How Variance Reduction Boosts A/B Test Sensitivity Without More Samples

This article explains why variance reduction is essential for A/B experiments, describes at‑assignment and post‑assignment techniques such as stratified sampling, post‑stratification and CUPED, compares their effectiveness, and presents real‑world case studies demonstrating how they improve experiment sensitivity without increasing sample size.

A/B testingCUPEDexperiment sensitivity

0 likes · 13 min read

How Variance Reduction Boosts A/B Test Sensitivity Without More Samples

Huolala Tech

Nov 17, 2023 · Fundamentals

Ensuring Homogeneity in AB Tests: Practical Solutions for Reliable Results

This article explains how to guarantee homogeneity in AB experiments by defining pre‑experiment bias, presenting statistical testing methods, outlining a three‑step workflow for both pre‑ and post‑experiment phases, and sharing real‑world case studies and correction techniques to improve decision‑making reliability.

AA testingAB testingCUPED

0 likes · 9 min read

Ensuring Homogeneity in AB Tests: Practical Solutions for Reliable Results

Tencent Advertising Technology

Jul 13, 2021 · Fundamentals

Experiment Design in Two‑Sided Markets – Key Insights from Tencent Advertising Live Session

In a July 8 live broadcast, Tencent Advertising’s strategy algorithm team explained experimental design for two‑sided markets, covering control‑variable selection, CUPED variance reduction, Bayesian smoothing, and bias metrics, and answered participant questions with practical examples and guidance.

CUPEDadvertising analyticsbayesian smoothing

0 likes · 4 min read

Experiment Design in Two‑Sided Markets – Key Insights from Tencent Advertising Live Session