Fundamentals of Statistics for A/B Testing and Its Application in the DeWu Platform

Effective A/B experiments rely on solid statistical theory. Understanding basic statistics helps design experiments, interpret results, and make data‑driven decisions.

1. Basic Statistical Concepts

Population and sample: the population is the entire group of interest (e.g., all DeWu app users); a sample is a randomly selected subset.

Sample size vs. sample count: sample count refers to the number of independent samples, while sample size refers to the number of observations within each sample.

Mean, variance, and expectation: the mean measures central tendency; variance quantifies dispersion; the expectation is the weighted average of possible outcomes.

Probability distributions: discrete (e.g., binomial) and continuous (e.g., normal) distributions are used to model metric behavior.

Central Limit Theorem (CLT) and Law of Large Numbers: with enough samples, the sampling distribution of the mean approaches a normal distribution, and sample averages converge to the true population mean.

2. Understanding A/B Testing

A/B testing compares two variants (A and B) on similar users to determine which performs better. It is a powerful tool for product optimization, risk reduction, and scientific validation.

3. Statistical Foundations in A/B Testing

3.1 Hypothesis Testing

Two competing hypotheses are set up: the null hypothesis (no difference between groups) and the alternative hypothesis (a significant difference exists). The test evaluates whether observed data are unlikely under the null hypothesis.

Significance level (α) and p‑value: a small p‑value (typically < 0.05) indicates that the observed difference is unlikely under the null hypothesis, leading to its rejection.

Confidence interval: provides a range of plausible values for the true effect size and helps assess significance.

4. DeWu’s A/B Platform

Features include pre‑allocation checks, idle‑period validation, health monitoring, metric management, self‑service data extraction, and one‑click report generation.

The platform integrates client‑side and server‑side experiments, supports SDK traffic splitting, and provides a data pipeline for metric calculation.

5. Summary

Typical A/B workflow: define goal → formulate hypothesis → design experiment → collect data → analyze results. Proper statistical understanding ensures reliable, data‑driven decisions while minimizing risk.