Why Your A/B Test Results Might Mislead You—and How to Interpret Them Correctly

Key statistical concepts in A/B testing

In an A/B test the p‑value is the probability of observing data at least as extreme as the actual result under the null hypothesis that the variation has no effect. When the p‑value is lower than a pre‑selected significance level (α, commonly 0.05 or 0.01) the result is called statistically significant . Two frequent misunderstandings are:

Significance ≠ importance – a result can be statistically significant while the practical effect size is negligible.

Non‑significant ≠ irrelevant – lack of significance may stem from insufficient sample size or a very small true effect.

Calculating the minimum sample size

The required sample size depends on four parameters:

Baseline conversion rate – the current metric value (e.g., 5 % click‑through). Extreme baselines need more observations.

Minimum Detectable Effect (MDE) – the smallest lift you aim to detect (e.g., 0.5 % absolute increase). Smaller MDE → larger sample.

Significance level (α) – stricter α (e.g., 0.01) increases the needed sample.

Statistical power (1‑β) – the probability of detecting a true effect; higher power requires more data.

A common approximation is:

n = ((Z_{1-α/2}+Z_{1-β})^2 * (p1(1-p1)+p2(1-p2))) / (p2-p1)^2

where p1 and p2 are the baseline and expected conversion rates. The article illustrates the intuition with a fishing‑bait example: when success rates are very low or very high, many trials are needed to distinguish a 1 % difference.

Experiment duration and stopping rules

Stop an experiment only after all three conditions are satisfied:

It spans a full user‑behavior cycle (e.g., at least one week for e‑commerce to capture weekday/weekend patterns).

Each variant reaches the pre‑calculated minimum sample size.

The statistical metric (p‑value) remains stable over a reasonable observation window; it should not fluctuate wildly.

Avoid “data peeking” – repeatedly checking results and terminating early when a metric appears significant – because it inflates the false‑positive rate.

Data‑interpretation pitfalls

Metric definition alignment : Designers, product managers and analysts must agree on the exact definition (e.g., whether “order conversion” includes only the landing page or the entire purchase funnel). Mis‑aligned definitions lead to systematic bias.

Segmentation and Simpson’s paradox : Aggregate results can mask opposite trends in sub‑populations. For example, a variant may appear worse overall but outperform the control for both male and female users separately. The article includes an illustration:

Novelty and primacy effects : New UI elements can temporarily boost engagement (novelty), while users may initially resist familiar flows that are changed (primacy). Mitigation strategies include extending the test period (e.g., ≥2 weeks) and analysing new vs. returning users separately.

Rational use of A/B results

An A/B test tells whether variation A or B performs better on the chosen metric, but it does not guarantee that the winning variant is the optimal solution overall. After a test, combine quantitative outcomes with qualitative insights (user interviews, session recordings) to understand the underlying behavior and to generate the next hypothesis.

Beware of over‑optimising for short‑term metrics at the expense of long‑term user experience, brand trust, or lifetime value.

E‑commerce conversion heuristics derived from experiments

Applying a “supply‑demand matching” mindset, the article recommends:

Show only core information (price, discount) as the primary hook.

Provide a clear reference frame (e.g., “N people liked this”).

Control information density: avoid repeating the same detail across many items.

Trim stable modules to free space for high‑impact content.

Maintain a consistent information structure across repeated units to reduce cognitive load.

Adjust density based on match accuracy – in high‑match scenarios use fewer, richer items; in low‑match scenarios increase the number of lighter items.