experiment design

JD Tech Talk

Jun 12, 2025 · Product Management

How to Tackle Outliers in Internet A/B Experiments: Methods, Pitfalls, and Practical Tips

This article explores why outliers appear in large‑scale internet A/B tests, explains their impact on experiment precision, compares traditional trim and winsorize techniques, reviews a range of statistical and machine‑learning detection methods, and offers practical recommendations for handling them in product experiments.

A/B testingexperiment designoutlier detection

0 likes · 15 min read

How to Tackle Outliers in Internet A/B Experiments: Methods, Pitfalls, and Practical Tips

Didi Tech

Apr 10, 2025 · Product Management

AA Testing and Rerandomization Techniques for Reliable AB Experiments

The article outlines how AA testing and rerandomization can detect and correct non‑uniform traffic splits in short‑term AB experiments, detailing three solutions—AA tests, seed‑based rerandomization, and retrospective AA analysis—along with theoretical guarantees, empirical error‑rate reductions, and remaining challenges for long‑term or clustered designs.

AA testingAB testingCUPED

0 likes · 17 min read

AA Testing and Rerandomization Techniques for Reliable AB Experiments

JD Tech

Jan 13, 2025 · Fundamentals

Handling Outliers in Internet A/B Experiments: Concepts, Methods, and Practical Recommendations

This article examines the challenges of outliers in large‑scale internet A/B testing, explains their statistical definition, outlines common causes, evaluates the benefits and limits of removal, and compares traditional trim and winsorize techniques along with practical detection and risk‑control strategies.

A/B testingTRIMdata analysis

0 likes · 8 min read

Handling Outliers in Internet A/B Experiments: Concepts, Methods, and Practical Recommendations

JD Retail Technology

Jan 7, 2025 · Fundamentals

Handling Outliers in Internet A/B Experiments: Concepts, Methods, and Practical Recommendations

The article explains why outliers destabilize internet A/B tests, outlines their causes, compares trimming and winsorizing, presents lightweight detection (e.g., kurtosis) and risk‑control strategies, and offers practical recommendations for bias‑aware removal and variance‑reduction techniques to improve experimental precision.

A/B testingBig DataTRIM

0 likes · 10 min read

Didi Tech

Dec 12, 2024 · Product Management

Key AB Testing Interview Questions and Answers for Data Science Candidates

The article reviews common AB‑testing interview questions for data‑science candidates, explaining the role of p‑values, Type I/II errors, the difference between statistical and business significance, why effects can vanish when scaling, and how to improve experiment sensitivity through larger samples, variance‑reduction methods, and careful metric design.

AB testingMDEexperiment design

0 likes · 12 min read

Key AB Testing Interview Questions and Answers for Data Science Candidates

DataFunTalk

Jul 22, 2024 · Fundamentals

A/B Testing and Causal Inference: Evolution of Sampling, Metric Evaluation, and Statistical Inference

The article reviews the development of online A/B testing, covering sampling and traffic‑splitting techniques, metric computation improvements, statistical inference advances, and current challenges such as interference, real‑time inference, and large‑scale metric computation, while referencing recent research papers.

A/B testingMetric Evaluationcausal inference

0 likes · 10 min read

A/B Testing and Causal Inference: Evolution of Sampling, Metric Evaluation, and Statistical Inference

DataFunTalk

May 23, 2024 · Fundamentals

Systematic Solutions to the AA Problem in Random Experiments

Speaker Wanbo Kui, a Didi data analyst, will present a systematic approach to addressing the AA problem in random experiments, covering academic and industry research on re-randomization, its principles and simulations, practical applications, and how it enhances experiment validity.

A/B testingAA problemexperiment design

0 likes · 3 min read

Systematic Solutions to the AA Problem in Random Experiments

DataFunTalk

May 22, 2024 · Fundamentals

Systematic Solutions to the AA Problem in Random Experiments

This presentation introduces the AA problem that can compromise A/B test validity and explains how combining re‑randomization with regression adjustment provides an effective, practical solution to improve experiment reliability and credibility.

A/B testingAA problemexperiment design

0 likes · 3 min read

DataFunSummit

Mar 10, 2024 · Artificial Intelligence

Evaluating Long-Term Strategy Effects with A/B Experiments: Causes, Industry Solutions, and Business Cases

This article examines why A/B experiments often capture only short‑term impacts, explains external and internal factors behind short‑ and long‑term effects, and presents seven industrial methods—including user‑learning models, personalized recommendation adjustments, surrogate metrics, and bias correction—supported by real‑world case studies.

A/B testingbias correctioncausal inference

0 likes · 14 min read

Evaluating Long-Term Strategy Effects with A/B Experiments: Causes, Industry Solutions, and Business Cases

Test Development Learning Exchange

Feb 2, 2024 · Product Management

Understanding AB Testing: Risks, Benefits, and Best Practices

AB testing is a statistical method for comparing multiple strategies or versions to determine the most effective one, and this article explains its risks, mitigation measures, advantages, key dimensions, step‑by‑step workflow, fairness considerations, data‑cleaning techniques, target goals, design guidelines, and alternative experimental approaches.

AB testingconversion ratedata analysis

0 likes · 9 min read

Understanding AB Testing: Risks, Benefits, and Best Practices

DataFunTalk

Feb 1, 2024 · Fundamentals

Understanding Search Experiments: AB Testing, Experiment Types, and Common Issues

This article explains search experiments from a data‑product viewpoint, covering AB testing fundamentals, multi‑layer experiment architecture, four experiment types (ordinary AB, vocabulary, diff‑AB, interleaving), real‑world case studies, and a comprehensive FAQ addressing typical challenges and troubleshooting methods.

A/B testingalgorithm evaluationdata product

0 likes · 10 min read

Understanding Search Experiments: AB Testing, Experiment Types, and Common Issues

DataFunTalk

Dec 20, 2023 · Fundamentals

Evaluating Long-Term Effects of Strategies with A/B Experiments: Methods and Case Studies

This article examines why A/B experiments often capture only short‑term impacts, categorises external and internal causes of short‑term bias, and presents seven industry‑tested approaches—including user‑learning models, personalized recommendation adjustments, surrogate metrics, and bias correction techniques—to reliably estimate long‑term strategy effectiveness, illustrated with real business cases.

A/B testingcausal inferenceexperiment design

0 likes · 13 min read

Evaluating Long-Term Effects of Strategies with A/B Experiments: Methods and Case Studies

DataFunTalk

Dec 14, 2023 · Fundamentals

Evaluating Long-Term vs Short-Term Effects in A/B Experiments

While A/B testing is widely used for data-driven decisions, short-term experimental results often diverge from long-term impacts, leading to misguided strategies; this article examines why such inconsistencies arise and reviews major methods—including extended experiments, holdout groups, post‑analysis, CCD, and surrogate‑metric modeling—to reliably estimate long‑term effects.

A/B testingLong-term impactMetrics

0 likes · 13 min read

Evaluating Long-Term vs Short-Term Effects in A/B Experiments

DataFunTalk

Dec 10, 2023 · Operations

Designing Experiments for Peak Surge Pricing in Two‑Sided Markets: Lessons from Uber, Lyft, DoorDash and Didi

This article examines how two‑sided platforms such as Uber, Lyft, DoorDash and Didi design and evaluate peak‑surcharge experiments, addressing network effects, bias‑variance trade‑offs, time‑space slicing, random‑saturation designs, and continuous bandit‑based testing within an operations‑focused experimental system.

AB testingcausal inferenceexperiment design

0 likes · 16 min read

Designing Experiments for Peak Surge Pricing in Two‑Sided Markets: Lessons from Uber, Lyft, DoorDash and Didi

DataFunSummit

Dec 4, 2023 · Product Management

Designing an AB Experiment System for User Growth Scenarios

This article presents a comprehensive AB testing framework tailored for new‑user growth scenarios, detailing the challenges of early traffic allocation, the scientific validation of a new experiment system, real‑world case studies, and practical guidelines for evaluation and implementation.

AB testingdata analysisexperiment design

0 likes · 14 min read

Designing an AB Experiment System for User Growth Scenarios

DataFunTalk

Nov 28, 2023 · Product Management

Challenges and Technical Solutions for Freight Bilateral Market Experiments

This article examines the unique challenges of conducting experiments in the freight bilateral market—covering transaction, pricing, marketing, and product scenarios—and presents a comprehensive technical solution framework that includes cluster traffic splitting, homogeneity assurance, efficient interpretation, and observational study methods.

bilateral marketcausal inferencedata science

0 likes · 12 min read

Challenges and Technical Solutions for Freight Bilateral Market Experiments

DataFunTalk

Sep 30, 2023 · Fundamentals

Different Types of Experiments in Search Scenarios

The presentation by Tencent PCG data product manager Wang Dongxing introduces A/B testing fundamentals and shares practical experiences with various search experiment methods—including regular A/B, vocabulary, diffAB, and interleaving—while highlighting common pitfalls and offering actionable insights for practitioners.

A/B testingdata productexperiment design

0 likes · 2 min read

Different Types of Experiments in Search Scenarios

DataFunTalk

Sep 27, 2023 · Product Management

Building an AB Experiment System for User Growth Scenarios

This article presents a comprehensive AB testing framework tailored for new‑user growth scenarios, detailing the challenges of early traffic splitting, the design of a scientifically validated experiment system, ID selection criteria, and real‑world case studies that demonstrate improved retention and device activation.

AB testingMobile Appsdata analysis

0 likes · 12 min read

Building an AB Experiment System for User Growth Scenarios

DataFunSummit

Sep 21, 2023 · Product Management

Avoiding Deceptive Conclusions in LinkedIn Advertising AB Tests and the Budget‑Splitting Method

This article explains how LinkedIn’s advertising teams prevent misleading AB‑test results, describes the challenges of large‑scale ad experiments such as cannibalization, reviews industry solutions, and introduces their innovative budget‑splitting experiment that dramatically improves statistical power.

AB testingLinkedInadvertising

0 likes · 15 min read

Avoiding Deceptive Conclusions in LinkedIn Advertising AB Tests and the Budget‑Splitting Method

DataFunTalk

Aug 3, 2023 · Game Development

Applying A/B Testing to Drive Growth in Tencent Overseas Games

This article explains how Tencent leverages A/B testing across its overseas games, detailing market differences, experimental methodology, multi‑cloud platform compliance, data architecture, and case studies that illustrate how targeted experiments improve user onboarding, gameplay settings, and email‑based re‑engagement.

A/B testingdata pipelinesexperiment design

0 likes · 12 min read

Applying A/B Testing to Drive Growth in Tencent Overseas Games