A/B testing | BestHub

JD Tech Talk

Jun 12, 2025 · Product Management

How to Tackle Outliers in Internet A/B Experiments: Methods, Pitfalls, and Practical Tips

This article explores why outliers appear in large‑scale internet A/B tests, explains their impact on experiment precision, compares traditional trim and winsorize techniques, reviews a range of statistical and machine‑learning detection methods, and offers practical recommendations for handling them in product experiments.

A/B testingexperiment designoutlier detection

0 likes · 15 min read

How to Tackle Outliers in Internet A/B Experiments: Methods, Pitfalls, and Practical Tips

ByteDance Data Platform

Apr 16, 2025 · Product Management

How A/B Testing Turns Guesswork into Data‑Driven Business Success

In today's fast‑changing market, traditional intuition‑based decisions falter, but systematic A/B testing—illustrated by ByteDance’s academic‑loop culture and real‑world case studies—empowers organizations to replace guesswork with evidence, accelerate innovation, and achieve measurable performance gains across products and strategies.

A/B testingdata-drivendecision making

0 likes · 18 min read

How A/B Testing Turns Guesswork into Data‑Driven Business Success

Dual-Track Product Journal

Feb 28, 2025 · Product Management

How to Turn Complex E‑commerce Promotion Logic into Drag‑and‑Drop Blocks

This article shows how a product manager transformed intricate promotion rules—like "spend 300, get 50 off" with exclusions—into a visual, low‑code builder using modular components, natural‑language parsing, safety checks, version control, A/B testing, and collaborative approval flows.

A/B testinge-commercelow-code

0 likes · 6 min read

How to Turn Complex E‑commerce Promotion Logic into Drag‑and‑Drop Blocks

ByteDance Data Platform

Feb 12, 2025 · Fundamentals

Why A/B Tests Fail in Recommendation Systems and How to Fix Them

This article examines the hidden complexities of A/B experiments in short‑video recommendation feeds, explains why traditional designs produce biased results due to learning, double‑sided, and network effects, and presents practical double‑sided and community‑randomized experiment frameworks to obtain unbiased strategy evaluations.

A/B testingCommunity randomizationDouble-sided effects

0 likes · 21 min read

Why A/B Tests Fail in Recommendation Systems and How to Fix Them

JD Tech

Jan 13, 2025 · Fundamentals

Handling Outliers in Internet A/B Experiments: Concepts, Methods, and Practical Recommendations

This article examines the challenges of outliers in large‑scale internet A/B testing, explains their statistical definition, outlines common causes, evaluates the benefits and limits of removal, and compares traditional trim and winsorize techniques along with practical detection and risk‑control strategies.

A/B testingTRIMdata analysis

0 likes · 8 min read

Handling Outliers in Internet A/B Experiments: Concepts, Methods, and Practical Recommendations

JD Retail Technology

Jan 7, 2025 · Fundamentals

Handling Outliers in Internet A/B Experiments: Concepts, Methods, and Practical Recommendations

The article explains why outliers destabilize internet A/B tests, outlines their causes, compares trimming and winsorizing, presents lightweight detection (e.g., kurtosis) and risk‑control strategies, and offers practical recommendations for bias‑aware removal and variance‑reduction techniques to improve experimental precision.

A/B testingTRIMbig data

0 likes · 10 min read

Model Perspective

Dec 2, 2024 · Fundamentals

What Is the Beta Distribution and Why It Matters in A/B Testing?

The Beta distribution is a flexible probability model defined on the interval [0,1] with two shape parameters that control its form, offering useful properties such as mean and variance, and is widely applied in A/B testing, risk assessment, and machine‑learning tasks to model proportions and uncertainties.

A/B testingbeta distributionmachine learning

0 likes · 5 min read

What Is the Beta Distribution and Why It Matters in A/B Testing?

ByteDance Data Platform

Jul 31, 2024 · Product Management

How Data‑Driven Flywheels Power User Growth: Insights from Volcengine

This article shares a data‑centric perspective on user growth, covering entropy reduction, information management, the data‑driven flywheel, A/B testing practices, retention strategies, and practical case studies that illustrate how systematic data analysis fuels sustainable product expansion.

A/B testingdata-drivenentropy reduction

0 likes · 16 min read

How Data‑Driven Flywheels Power User Growth: Insights from Volcengine

DataFunTalk

Jul 22, 2024 · Fundamentals

A/B Testing and Causal Inference: Evolution of Sampling, Metric Evaluation, and Statistical Inference

The article reviews the development of online A/B testing, covering sampling and traffic‑splitting techniques, metric computation improvements, statistical inference advances, and current challenges such as interference, real‑time inference, and large‑scale metric computation, while referencing recent research papers.

A/B testingMetric Evaluationcausal inference

0 likes · 10 min read

A/B Testing and Causal Inference: Evolution of Sampling, Metric Evaluation, and Statistical Inference

DataFunSummit

Jun 22, 2024 · Artificial Intelligence

Applying Causal Inference and Uplift Modeling for User Growth: Concepts, Methods, and Practice

This article introduces causal inference fundamentals, distinguishes correlation from causation, reviews major methodological streams, and demonstrates how uplift and gain models—implemented with T‑learner, S‑learner, and tree‑based approaches—can be applied to user growth and marketing scenarios, including evaluation metrics and future challenges.

A/B testingUplift Modelingcausal inference

0 likes · 14 min read

Applying Causal Inference and Uplift Modeling for User Growth: Concepts, Methods, and Practice

DataFunTalk

Jun 13, 2024 · Artificial Intelligence

A/B Testing and Model Grayscale in Credit Risk Control: Concepts, Requirements, and Integrated Solutions

This article explains how A/B testing and model grayscale are applied in credit risk control, discusses the specific requirements for effective testing, compares upstream and risk‑system traffic splitting methods, and proposes an integrated all‑in‑one solution to simplify feature engineering, model evaluation, and deployment.

A/B testingcredit riskfeature engineering

0 likes · 5 min read

A/B Testing and Model Grayscale in Credit Risk Control: Concepts, Requirements, and Integrated Solutions

DataFunTalk

May 25, 2024 · Fundamentals

Systematic Solutions to the AA Problem in Random Experiments

This talk explains how combining heavy randomization with regression adjustment can effectively mitigate AA problems in A/B testing, improving experiment credibility by addressing covariate imbalance and enhancing result validity for data‑driven decision making.

A/B testingAA problemExperimental design

0 likes · 2 min read

Systematic Solutions to the AA Problem in Random Experiments

DataFunTalk

May 23, 2024 · Fundamentals

Systematic Solutions to the AA Problem in Random Experiments

Speaker Wanbo Kui, a Didi data analyst, will present a systematic approach to addressing the AA problem in random experiments, covering academic and industry research on re-randomization, its principles and simulations, practical applications, and how it enhances experiment validity.

A/B testingAA problemexperiment design

0 likes · 3 min read

DataFunTalk

May 22, 2024 · Fundamentals

Systematic Solutions to the AA Problem in Random Experiments

This presentation introduces the AA problem that can compromise A/B test validity and explains how combining re‑randomization with regression adjustment provides an effective, practical solution to improve experiment reliability and credibility.

A/B testingAA problemexperiment design

0 likes · 3 min read

DataFunSummit

May 12, 2024 · Artificial Intelligence

Pairwise Data Based A/B Experiments: Unbiased Causal Inference in Network Experiments

The DataFun Data Science Summit on May 25 will feature Tencent data scientist Li Yilin presenting a comprehensive talk on pairwise‑data A/B experiments, covering unbiased estimation under various randomizations, theoretical analysis, and practical insights for causal inference in network‑driven online experiments.

A/B testingcausal inferencedata science

0 likes · 4 min read

Pairwise Data Based A/B Experiments: Unbiased Causal Inference in Network Experiments

DataFunTalk

May 12, 2024 · Artificial Intelligence

Paired Data Based A/B Experiments: Causal Inference in Network Experiments

The DataFun Data Science Summit on May 25 will feature Tencent data scientist Li Yilin presenting a comprehensive overview of paired‑data A/B experiments, covering causal inference challenges, unbiased estimators under various randomization designs, theoretical analysis, and practical insights for network‑based online experiments.

A/B testingcausal inferencenetwork experiments

0 likes · 5 min read

Paired Data Based A/B Experiments: Causal Inference in Network Experiments

DataFunSummit

May 9, 2024 · Fundamentals

Technical Evolution and Challenges of Online A/B Testing

This article reviews the two‑decade evolution of online A/B testing, outlines the business and technical challenges enterprises face, and details three core technical challenges—experiment accuracy, analysis & interpretation, and efficiency—along with practical solutions for each.

A/B testingEfficiencyanalysis

0 likes · 6 min read

Technical Evolution and Challenges of Online A/B Testing

ByteDance Data Platform

May 8, 2024 · Backend Development

How DataTester’s Architecture Upgrade Uses DDD to Tame Code Complexity

DataTester’s A/B testing platform underwent a comprehensive architectural overhaul, applying domain‑driven design, modular refactoring, automated validation, and dependency inversion to reduce change amplification, cognitive load, and unknown unknowns, ultimately improving code readability, maintainability, scalability, and development efficiency across its lifecycle.

A/B testingDDDbackend development

0 likes · 29 min read

How DataTester’s Architecture Upgrade Uses DDD to Tame Code Complexity

NetEase Cloud Music Tech Team

Apr 23, 2024 · Mobile Development

Cloud Music User Push Notification Optimization: Practices and Insights

Cloud Music revamped its push‑notification system by separating business and channel layers, integrating a unified delivery platform, tailoring messages to Android manufacturers, adding new push channels, refining frequency and copy controls, and using AI‑generated creatives, which together doubled click‑through rates and nearly doubled total click users within two months.

A/B testingAIGC Content GenerationCloud Music

0 likes · 23 min read

Cloud Music User Push Notification Optimization: Practices and Insights

DataFunSummit

Mar 10, 2024 · Artificial Intelligence

Evaluating Long-Term Strategy Effects with A/B Experiments: Causes, Industry Solutions, and Business Cases

This article examines why A/B experiments often capture only short‑term impacts, explains external and internal factors behind short‑ and long‑term effects, and presents seven industrial methods—including user‑learning models, personalized recommendation adjustments, surrogate metrics, and bias correction—supported by real‑world case studies.

A/B testingbias correctioncausal inference

0 likes · 14 min read

Evaluating Long-Term Strategy Effects with A/B Experiments: Causes, Industry Solutions, and Business Cases