AA Testing and Rerandomization Techniques for Reliable AB Experiments

The article outlines how AA testing and rerandomization can detect and correct non‑uniform traffic splits in short‑term AB experiments, detailing three solutions—AA tests, seed‑based rerandomization, and retrospective AA analysis—along with theoretical guarantees, empirical error‑rate reductions, and remaining challenges for long‑term or clustered designs.

AA testingAB testingCUPED

0 likes · 17 min read

AA Testing and Rerandomization Techniques for Reliable AB Experiments

Huolala Tech

Feb 2, 2024 · Fundamentals

How the Delta Method Improves AB Test Variance Estimation When Units Differ

This article explains why traditional hypothesis‑testing methods can mis‑estimate variance when the splitting unit and analysis unit differ in AB experiments, introduces the Delta Method as an unbiased variance estimator, compares it with Bootstrap and other corrections through simulations and real‑world case studies, and highlights its computational efficiency.

AB testingBootstrapDelta Method

0 likes · 8 min read

How the Delta Method Improves AB Test Variance Estimation When Units Differ