This article explores why outliers destabilize online A/B tests, explains their statistical definitions, compares trimming and winsorizing techniques, reviews classic and machine‑learning detection methods, and offers practical guidance for applying these approaches to improve experiment reliability.
This article, the fourth in the Trusted Experiment Whitepaper series, explains coin‑flip, complete, and paired random rotation designs, their grouping mechanisms, evaluation principles, and how to handle spillover and carryover effects to improve the reliability and power of AB experiments.
This article examines the hidden complexities of A/B experiments in short‑video recommendation feeds, explains why traditional designs produce biased results due to learning, double‑sided, and network effects, and presents practical double‑sided and community‑randomized experiment frameworks to obtain unbiased strategy evaluations.
This talk explains how combining heavy randomization with regression adjustment can effectively mitigate AA problems in A/B testing, improving experiment credibility by addressing covariate imbalance and enhancing result validity for data‑driven decision making.
Randomized Controlled Trials (RCTs) are considered the gold standard for establishing causal relationships because randomization balances known and unknown confounders, control groups provide clear comparisons, and reproducibility ensures reliable results, though practical limitations like cost and ethics often require alternative observational methods.
This article explains the statistical foundations of A/B testing, introduces potential outcomes and average treatment effect, defines metric covariance, and presents practical estimation methods—including naive, data‑augmentation, and bucket‑based approaches—along with real‑world performance evaluations and applications such as variance reduction and Bayesian optimization.
This article explains the fundamentals of causal inference, illustrates its distinction from correlation modeling, and demonstrates how causal techniques can be applied to limited‑resource decision problems such as knapsack optimization, ride‑hailing subsidies, and flight pricing, while also covering experimental design, popular models, evaluation metrics, and open challenges.
This article discusses experimental design challenges in two-sided markets, particularly in advertising scenarios, and presents various methods including four-table experiments, counterfactual interleaving, and contingency table joint sampling to address issues like network effects and competition between supply and demand sides.
This article presents a practical example using SPSS to perform an independent‑samples t‑test on iron concentrations in mouse livers from two diet groups, explains the assumptions, walks through data entry, test configuration, result interpretation, and concludes that the dietary effect is not statistically significant.
This article explains why randomised experiments are the gold standard for turning correlations into causal claims, illustrates their use in evaluating online versus face‑to‑face learning, and discusses ideal experimental design, assignment mechanisms, and key take‑aways for causal inference.
This article explains the theory behind two‑factor ANOVA, distinguishes cases with and without interaction, presents the mathematical model, and demonstrates a complete Python implementation using statsmodels, including data setup, model fitting, and interpretation of the ANOVA table.
The paper presents a comprehensive A/B testing framework for online experiments that guides practitioners through four stages—designing objectives and metrics, implementing random traffic allocation with robustness checks, evaluating effects using descriptive statistics and hypothesis testing, and making rollout decisions based on multidimensional significance and attribution analyses.
This article explains how data analysts can use experimental and observational research methods, including randomized controlled trials, quasi‑experiments, difference‑in‑differences, regression discontinuity, synthetic control, and Bayesian structural time‑series, to assess the causal impact of product and marketing changes on business metrics.
This comprehensive guide explains AB testing fundamentals—from defining control and experimental groups and avoiding confounding factors, to calculating sample size, selecting ratio‑based metrics, tracking data, monitoring experiments, and analyzing statistical significance—providing a step‑by‑step methodology for data‑driven product optimization.
The article explains how data products create business value through application, data, and algorithm layers, using examples like 5G infrared temperature screening and ImageNet, and discusses the roles of experimental design, causal inference, and reinforcement learning in building effective AI‑driven strategies.
This article presents a series of data‑science questions and expert answers covering overfitting, experimental design for user behavior, the distinction between tall and wide data, detecting chart junk, outlier detection methods, extreme‑value theory for rare events, recommendation‑engine fundamentals, and techniques for visualizing high‑dimensional data.
