Statistical Foundations and Practical Applications of A/B Testing

A/B testing is a data‑driven experimental method that splits traffic to compare different product versions, relying on statistical analysis to ensure scientific and accurate decision making.

The core requirements include similarity and uniformity of the experimental population, adherence to the single‑variable principle, and rigorous effect evaluation.

Statistical foundations involve defining the population (all users), samples (representative subsets), parameters (population metrics such as mean), and statistics (sample metrics). Key concepts such as mean, variance, and normal distribution underpin hypothesis testing.

Hypothesis testing formulates a null hypothesis (H0) and an alternative hypothesis (H1). Errors of the first kind (α) and second kind (β) are controlled, with a typical significance level of 0.05 and power of 0.8.

Significance is assessed via p‑values; if p ≤ 0.05 the null hypothesis is rejected, indicating a statistically significant difference between variants.

Common tests include z‑test, t‑test, and chi‑square test. For large samples, an independent two‑sample t‑test is often used, calculating the test statistic from sample means, standard deviations, and sizes.

Metrics are categorized as rate metrics (Bernoulli‑distributed) and mean metrics (Gaussian‑distributed), each with specific p‑value calculation formulas. Composite metrics (e.g., conversion rate) require careful handling of denominators.

An example from a mobile app experiment compares two popup designs, using sample sizes and observed click‑through rates to compute a p‑value of 0.0329, which is below the 0.05 threshold, confirming the superiority of the test variant.

In summary, A/B testing provides a scientific, user‑centric approach to product decision making, improving efficiency and reducing the risk of suboptimal choices across various business scenarios.