Experimental design

ByteDance Data Platform

Feb 12, 2025 · Fundamentals

Why A/B Tests Fail in Recommendation Systems and How to Fix Them

This article examines the hidden complexities of A/B experiments in short‑video recommendation feeds, explains why traditional designs produce biased results due to learning, double‑sided, and network effects, and presents practical double‑sided and community‑randomized experiment frameworks to obtain unbiased strategy evaluations.

A/B testingCommunity randomizationDouble-sided effects

0 likes · 21 min read

Why A/B Tests Fail in Recommendation Systems and How to Fix Them

DataFunTalk

May 25, 2024 · Fundamentals

Systematic Solutions to the AA Problem in Random Experiments

This talk explains how combining heavy randomization with regression adjustment can effectively mitigate AA problems in A/B testing, improving experiment credibility by addressing covariate imbalance and enhancing result validity for data‑driven decision making.

A/B testingAA problemExperimental design

0 likes · 2 min read

Systematic Solutions to the AA Problem in Random Experiments

Model Perspective

Jan 3, 2024 · Fundamentals

How Randomized Controlled Trials Reveal True Causality

Randomized Controlled Trials (RCTs) are considered the gold standard for establishing causal relationships because randomization balances known and unknown confounders, control groups provide clear comparisons, and reproducibility ensures reliable results, though practical limitations like cost and ethics often require alternative observational methods.

Experimental designcausal inferencerandomized controlled trials

0 likes · 7 min read

How Randomized Controlled Trials Reveal True Causality

DataFunSummit

Apr 22, 2023 · Artificial Intelligence

Applying Causal Inference to Limited‑Resource Decision‑Making

This article explains the fundamentals of causal inference, illustrates its distinction from correlation modeling, and demonstrates how causal techniques can be applied to limited‑resource decision problems such as knapsack optimization, ride‑hailing subsidies, and flight pricing, while also covering experimental design, popular models, evaluation metrics, and open challenges.

Experimental designcausal inferencedecision optimization

0 likes · 15 min read

Applying Causal Inference to Limited‑Resource Decision‑Making

Tencent Advertising Technology

Mar 28, 2023 · Operations

Experimental Design for Two-Sided Markets in Advertising Scenarios

This article discusses experimental design challenges in two-sided markets, particularly in advertising scenarios, and presents various methods including four-table experiments, counterfactual interleaving, and contingency table joint sampling to address issues like network effects and competition between supply and demand sides.

A/B testingExperimental designNetwork effects

0 likes · 14 min read

Experimental Design for Two-Sided Markets in Advertising Scenarios

Model Perspective

Dec 7, 2022 · Fundamentals

Can Different Diets Change Mouse Liver Iron? A Step‑by‑Step SPSS t‑Test Guide

This article presents a practical example using SPSS to perform an independent‑samples t‑test on iron concentrations in mouse livers from two diet groups, explains the assumptions, walks through data entry, test configuration, result interpretation, and concludes that the dietary effect is not statistically significant.

Experimental designSPSSbiostatistics

0 likes · 4 min read

Can Different Diets Change Mouse Liver Iron? A Step‑by‑Step SPSS t‑Test Guide

Model Perspective

Sep 9, 2022 · Fundamentals

How Random Experiments Reveal True Causal Effects in Education

This article explains why randomised experiments are the gold standard for turning correlations into causal claims, illustrates their use in evaluating online versus face‑to‑face learning, and discusses ideal experimental design, assignment mechanisms, and key take‑aways for causal inference.

Experimental designcausal inferenceeducation

0 likes · 10 min read

How Random Experiments Reveal True Causal Effects in Education

Model Perspective

Jul 15, 2022 · Fundamentals

How to Perform Two-Way ANOVA with Python’s statsmodels: Theory and Code

This article explains the theory behind two‑factor ANOVA, distinguishes cases with and without interaction, presents the mathematical model, and demonstrates a complete Python implementation using statsmodels, including data setup, model fitting, and interpretation of the ANOVA table.

Experimental designpythonstatistical analysis

0 likes · 6 min read

How to Perform Two-Way ANOVA with Python’s statsmodels: Theory and Code

Alimama Tech

Jul 14, 2021 · Big Data

A/B Testing Framework for Online Experiments: Design, Implementation, Analysis, and Decision Making

The paper presents a comprehensive A/B testing framework for online experiments that guides practitioners through four stages—designing objectives and metrics, implementing random traffic allocation with robustness checks, evaluating effects using descriptive statistics and hypothesis testing, and making rollout decisions based on multidimensional significance and attribution analyses.

A/B testingExperimental designdata analysis

0 likes · 22 min read

A/B Testing Framework for Online Experiments: Design, Implementation, Analysis, and Decision Making

Liulishuo Tech Team

Oct 26, 2020 · Fundamentals

Causal Inference Methods for Quantifying Product Impact in Data Analytics

This article explains how data analysts can use experimental and observational research methods, including randomized controlled trials, quasi‑experiments, difference‑in‑differences, regression discontinuity, synthetic control, and Bayesian structural time‑series, to assess the causal impact of product and marketing changes on business metrics.

AB testingDifference-in-DifferencesExperimental design

0 likes · 7 min read

Causal Inference Methods for Quantifying Product Impact in Data Analytics

vivo Internet Technology

Jun 3, 2020 · Product Management

A Comprehensive Guide to AB Testing: Methodology and Implementation

This comprehensive guide explains AB testing fundamentals—from defining control and experimental groups and avoiding confounding factors, to calculating sample size, selecting ratio‑based metrics, tracking data, monitoring experiments, and analyzing statistical significance—providing a step‑by‑step methodology for data‑driven product optimization.

A/B Testing MethodologyAB testingExperimental design

0 likes · 14 min read

A Comprehensive Guide to AB Testing: Methodology and Implementation

DataFunTalk

Mar 27, 2020 · Artificial Intelligence

Understanding Data Product Layers: Business Value, Data, Algorithms, and Applications

The article explains how data products create business value through application, data, and algorithm layers, using examples like 5G infrared temperature screening and ImageNet, and discusses the roles of experimental design, causal inference, and reinforcement learning in building effective AI‑driven strategies.

Experimental designartificial intelligencebusiness value

0 likes · 8 min read

Understanding Data Product Layers: Business Value, Data, Algorithms, and Applications

Architects Research Society

Nov 21, 2016 · Artificial Intelligence

Data Science Q&A: Overfitting, Experimental Design, Tall/Wide Data, Chart Junk, Outliers, Extreme Value Theory, Recommendation Engines, and Visualization

This article presents a series of data‑science questions and expert answers covering overfitting, experimental design for user behavior, the distinction between tall and wide data, detecting chart junk, outlier detection methods, extreme‑value theory for rare events, recommendation‑engine fundamentals, and techniques for visualizing high‑dimensional data.

Experimental designRecommendation systemschart junk

0 likes · 18 min read

Data Science Q&A: Overfitting, Experimental Design, Tall/Wide Data, Chart Junk, Outliers, Extreme Value Theory, Recommendation Engines, and Visualization