This article shares practical experiences and insights on building an experiment platform, covering A/B testing applications, platform selection, broader business use cases, rapid system integration, and robust experiment analysis techniques such as pre‑AA checks, SRM, Welch’s T and Z tests.
This article details the technical design, challenges, and results of Taobao’s “Home‑Page as Venue” initiative, explaining how a new A/B‑page architecture replaced crowded promotional cards with a dynamic cat‑head entry, improved information‑flow efficiency, reduced development cost, and delivered measurable gains in traffic, conversion, and revenue during major sales events.
This article demonstrates how to compare the distribution of a variable across control and treatment groups using Python, covering data generation, visual techniques such as boxplots, histograms, KDE, CDF, QQ and ridgeline plots, and a suite of statistical tests including t‑test, SMD, Mann‑Whitney, permutation, chi‑square, Kolmogorov‑Smirnov and ANOVA for both two‑group and multi‑group scenarios.
DataTester’s Open Platform provides a configurable, extensible A/B testing solution that integrates via OpenAPI, SDKs, and a visual integration console, enabling enterprises to customize experiments, reduce development costs, and streamline deployment while supporting diverse business needs through modular components and a material ecosystem.
This article explains the fundamentals of A/B testing, why it is essential for data‑driven product decisions, how to design and run experiments—including hypothesis formulation, metric selection, sample size calculation, traffic segmentation, and duration planning—and how to analyze results using T‑tests, P‑values, and structured decision processes.
This article chronicles the development, architectural challenges, and engineering solutions of a large‑scale game recommendation platform, covering background, initial models, business growth, caching strategies, GC optimization, rate‑limiting, fine‑grained operations, multi‑path recall, A/B testing, and future intelligent enhancements.
This article demonstrates how to compare the distribution of a variable across treatment and control groups using Python, covering data simulation, visual techniques such as boxplots, histograms, KDE, CDF, QQ and ridgeline plots, and statistical tests including t‑test, SMD, Mann‑Whitney, permutation, chi‑square, KS and ANOVA.
This article provides a comprehensive overview of causal inference, explaining its definition, the distinction between correlation and causation, classic pitfalls such as Simpson's paradox, key metrics like ATE and ATT, experimental designs, bias mitigation techniques, and practical case studies from content platforms and the Titanic dataset.
The article details the design, evolution, and operational practices of Vivo’s large‑scale game recommendation platform, covering its initial rule‑based model, layered strategy framework, multi‑level caching, GC tuning, rate‑limiting, fine‑grained A/B testing, multi‑path recall, dynamic exposure control, and future intelligent extensions.
This article explains how to design and evaluate experiments for bilateral markets in advertising platforms, covering the limitations of traditional randomization, the four‑cell traffic‑advertisement experiment, various mitigation strategies such as counterfactual interleaving and joint sampling, and the use of a simulation system to validate methods.
This article explains the scientific foundations, historical origins, statistical principles, implementation techniques, and practical applications of A/B testing as a data‑driven growth tool for product optimization, algorithm iteration, and marketing decisions in modern internet companies.
After attending the DataFun causal inference summit, this article outlines why causal analysis matters, its typical use cases, practical challenges, its relationship with A/B testing, and how it integrates with machine learning and deep learning to improve decision‑making and model robustness.
A/B testing is a scientific method for product iteration that uses random user grouping, traffic segmentation, and metric analysis to derive representative conclusions, widely applied across major tech companies for evaluating ROI, with detailed workflow, example scenarios, and guidance on design and analysis.
This article describes how DataTester, Volcano Engine's A/B testing platform, achieved over four‑fold query speedup by applying five technical optimizations—including pre‑aggregation, join reduction, GroupBy redesign, AU‑metric caching, and asynchronous query handling—targeting both data construction and execution layers.
This article discusses how to improve the reliability of online A/B experiments by designing robust architecture, evaluating group balance with orthogonal testing, ensuring consistent front‑end/back‑end logging, automating statistical significance checks, reducing group imbalance, and interpreting results using causal trees.
This article details the private‑deployment architecture of Volcano Engine A/B Testing (DataTester), outlines three major challenges—version management, performance optimization, and stability—and explains the branch‑logic, release pipeline, model‑optimization, and pre‑aggregation solutions implemented to enable reliable, low‑resource SaaS‑like operation in on‑premise clusters.
This article details Volcano Engine DataTester's private deployment architecture, the version‑management, performance, and stability challenges encountered, and the business‑oriented solutions—including branch strategies, pipeline automation, ClickHouse model optimizations, and multi‑level caching—that enable reliable, efficient A/B testing in on‑premise environments.
This article reviews the two main streams of causal inference—potential‑outcome (Rubin) models and structural causal (Pearl) diagrams—covers classic techniques such as A/B testing, instrumental variables, matching, difference‑in‑differences, synthetic controls, matrix completion, heterogeneous treatment effect estimation, and discusses modern machine‑learning‑based approaches and causal discovery algorithms.
This article explores essential microservice traffic management techniques—including canary releases, A/B testing, and traffic shading—detailing their value, implementation steps, and practical examples using Service Mesh and Istio, with code snippets and diagrams to illustrate routing based on request attributes.
This article details a series of design experiments on 58’s homepage feed cards, analyzing how image realism, benefit labeling, and location presentation affect user click-through rates, revealing that authentic visuals, precise benefit information, and intuitive positioning outperform generic designs, and outlines a cyclical iteration process for continuous improvement.
This article presents a detailed case study of how a video app team used two rounds of A/B testing with different swipe‑up guide designs to diagnose retention issues, refine the user onboarding experience, and ultimately achieve significant improvements in new‑user retention and engagement metrics.
This article analyzes the performance bottlenecks of A/B experiment report queries on Apache Doris at Xiaomi, presents data-driven insights on query latency, field usage, and experiment ID matching, and details a series of optimizations—including pre‑aggregation, materialized views, bitmap deduplication, and schema redesign—that reduced query times by up to 60× and lowered cluster load.
This article examines how data‑driven design experiments—like Google’s 41‑shade blue link test and other color A/B studies—reveal both the power and pitfalls of relying solely on metrics, urging designers to balance data with intuition, broader context, and thoughtful KPI selection.
This article presents a data‑driven operational design case study for 58APP, detailing a five‑step testing framework, A/B experiments on UI elements such as real versus 3D personas and copy versus graphics, and the resulting insights that significantly improved click‑through and conversion rates.
The article presents the design, scientific methodology, system architecture, and case studies of Yanxuan's A/B testing platform, detailing how statistical principles, automated tracking, traffic allocation models, and unified reporting accelerate decision‑making and reduce development effort in e‑commerce experiments.
The article examines why classic A/B testing frameworks struggle with modern internet services—highlighting issues of intervention, measurement, and analysis—while proposing an observational, dynamic, and decision‑oriented next‑generation experiment system that leverages statistical learning and Bayesian optimization.
This article presents a comprehensive overview of Volcano Engine's A/B testing platform, detailing its four core stages—reliable experiment system, efficient data construction, scientific statistical analysis, and fine-grained governance—while explaining execution components, data pipelines, statistical methods, and operational best practices for large‑scale experimentation.
The article describes Tubi’s internal experimentation platform, explaining how traditional user‑group A/B tests can suffer from network interference and how Switchback experiments—time‑window based designs—address these issues, detailing their implementation, statistical methods, and the practical challenges overcome.
The article introduces the business background, architectural design, evolution challenges, and step‑by‑step methodology for building and operating a risk‑control experiment platform that supports online and offline A/B testing, data analysis, and strategy iteration in internet finance.
After more than a year of effort, Uber migrated its entire experimentation and feature‑flag ecosystem—including thousands of developers, dozens of partners, multiple mobile apps, and hundreds of services—to a new, unified platform that improves reliability, flexibility, and data quality while retiring over 50,000 legacy experiments.
This article explains common pain points in A/B testing, introduces essential statistical concepts such as sampling distribution, parameter estimation, confidence intervals, and hypothesis testing, and provides step‑by‑step formulas and a concrete example for calculating the minimum sample size needed to run a trustworthy experiment.
A solid grasp of basic statistical concepts—such as populations, samples, means, variance, probability distributions, the Central Limit Theorem, and hypothesis testing—enables designers of A/B experiments to correctly size samples, interpret p‑values and confidence intervals, and reliably deploy DeWu’s integrated platform for automated experiment allocation, metric monitoring, and one‑click reporting, ultimately driving data‑driven product decisions.
This guide explains how to use Ingress-Nginx's canary annotations for blue‑green, weight‑based, header‑based, and cookie‑based traffic splitting in Kubernetes, providing step‑by‑step YAML examples, command‑line verification, and a comparison of deployment strategies.
Huolala’s A/B testing platform, serving over 200 business scenarios and thousands of experiments, combines a three‑stage workflow with a modular architecture—including a configuration console, SDK for traffic routing and data collection, and a robust data‑recovery pipeline—while offering five distinct allocation algorithms to ensure scientific experiment results.
This article explains the origins of online A/B testing, outlines three typical product scenarios—recommendation, operations, and UI/UX—and presents classic case studies from companies like Bing, Google, Amazon, and TikTok to illustrate how controlled experiments drive data‑driven product optimization and measurable business impact.
This article explains the concept of conversion rate, its significance for product success, and presents a systematic, full‑process approach to identify and optimize key decision points—covering user and scenario analysis, metric definition, business and user research, and concrete design strategies for entry, detail, and submission stages.
This article explains the fundamentals of A/B testing, how Netflix designs experiments with random sampling and assignment, evaluates metrics to infer causal impact, and uses these insights to guide product decisions and improve member engagement.
This article explains how Netflix applies large‑scale A/B experiments and scientific methods to make data‑driven product decisions, describing its decision‑making frameworks, the role of hypothesis testing, and the upcoming topics in the series on experimentation.
This article explains how the Picasso platform provides a standardized, high‑efficiency A/B testing pipeline for front‑end developers, covering traffic allocation algorithms, flow reuse, orthogonal and mutually exclusive experiments, multi‑scenario rules, custom metrics, reporting, and future enhancements.
This article presents Vivo Game Center's end‑to‑end intelligent gray‑release data system, detailing its experimental mindset, statistical methods, data models, and product solutions that ensure scientific version evaluation, project progress, and rapid issue closure through root‑cause analysis and full‑process automation.
This article details Vivo Game Center's end‑to‑end intelligent gray‑release data framework—covering experiment design, statistical methods, data models, and automated product solutions—to ensure scientific version evaluation, accelerate project timelines, and quickly close the gray‑testing loop.
Repeating the same A/B test inflates false‑positive rates, creates inconsistent results, and hampers decision‑making, so a single well‑designed experiment with proper metrics, traffic allocation, analysis, and decision steps is essential for reliable product evaluation.
This article explains why designers must master data analysis, defines what “design data analysis” means, and walks through a three‑step framework—data splitting, tracking, and analysis—illustrated with practical e‑commerce and recruitment case studies to boost product metrics and retention.
The paper demonstrates how self‑supervised models SwAV and M6 generate high‑quality image and multimodal embeddings for Meiping Meiwu’s recommendation system, delivering notable gains in scene/style consistency, ranking AUC, classification and retrieval performance, especially for cold‑start items, and achieving measurable production lifts.
The DataFun Summit 2022 features an Experimental Science and Causal Inference forum where leading data scientists from Didi, Tencent, Google, ByteDance, and others present deep technical talks on causal inference methods, A/B testing, game operations, and advertising experiments, offering practical insights and audience takeaways.
This article details a year‑long series of experiments on the Taobao “Every Square Every House” infinite‑scroll light app, describing how added recall paths, a coarse‑ranking filter, multi‑task MMOE sorting, a lightweight down‑scroll predictor, and relevance‑enhanced features together boosted click‑through, scroll depth and per‑user engagement by double‑digit percentages.
This article analyzes how a home‑cleaning platform increased Chinese New Year service orders by designing a multi‑service ordering flow, comparing four packaging options, selecting a package‑recommendation model, eliminating three key friction points, and validating the solution through A/B testing and metric improvements.
This article introduces the A/B testing platform used at AutoHome, detailing its architecture, experiment flow, traffic allocation strategies, and statistical evaluation techniques such as hypothesis testing, confidence intervals, and significance testing, to guide data‑driven decision making for recommendation system improvements.
This paper introduces a novel multi-dimensional causal forest model combined with efficient integer programming algorithms to estimate heterogeneous treatment effects (HTE) in marketing scenarios, outperforming traditional tree-based methods through improved handling of intervention heterogeneity and resource allocation optimization.
This article explains why A/B testing is essential for data‑driven decision making, describes the Volcano Engine A/B testing system architecture, outlines practical experiment design, statistical analysis methods, real‑world case studies, and forecasts industry and technical trends for the practice.
Taobao’s senior‑friendly “Elder Mode” redesign tackles the challenges of an aging population by enlarging fonts, simplifying information, adding voice search, creating a walking‑based game, and implementing seamless version switching, all backed by extensive user research, technical solutions, and A/B experiments to improve the digital shopping experience for older users.
The article explains common release strategies—including blue‑green deployment, rolling updates, canary (gray) releases, and A/B testing—detailing their principles, advantages, limitations, and practical considerations for safely delivering new versions in production environments.
This article explains the wide applicability of causal inference, outlines three major challenges—correlation vs. causation, confounding factors, and selection bias—and demonstrates a scientific three‑level approach using examples such as smoking and a real‑world deployment of an intelligent client‑management tool at Beike, including experimental designs, results, and lessons learned.
This article explains why A/B testing is crucial for data‑driven product decisions, outlines ByteDance’s A/B testing system architecture across multiple layers, describes client‑ and server‑side experiment workflows, shares statistical best practices, and presents real‑world case studies illustrating hypothesis generation, evaluation, and future industry trends.
This article explains the concept of gray release, outlines a simple architecture with essential components, describes common traffic-splitting strategies, shows how to implement control in Nginx and service layers, and discusses complex scenarios such as multi‑service and data‑centric deployments.
This article reviews the main deployment techniques—blue‑green, rolling, canary (gray) releases and A/B testing—explaining their principles, advantages, drawbacks, and practical considerations for safely delivering new versions during iterative project development.
This article examines how data‑driven design decisions across header images, interactive floors, and template optimizations during the 11.11 shopping event dramatically increased click‑through rates, revealing actionable insights for commercial value, user experience, and conversion optimization.
This article explains common deployment strategies—including blue‑green, rolling, canary (gray) releases, and A/B testing—detailing their principles, advantages, drawbacks, and practical considerations for safely delivering new versions in production environments.
This article compares key deployment strategies—blue‑green, rolling, canary (gray) releases, and A/B testing—explaining their workflows, advantages, limitations, and practical considerations such as risk mitigation, resource usage, rollback challenges, and infrastructure requirements for modern software projects.
This article presents a comprehensive case study of applying edge AI to video preloading on the Xigua Video platform, detailing scenario analysis, predictive modeling of user behavior, feature engineering, on‑device model inference, dynamic algorithm package deployment, experimental evaluation, and the resulting performance and cost improvements.
The article explains four major software release strategies—blue‑green deployment, rolling release, canary (gray) release, and A/B testing—detailing their principles, advantages, drawbacks, and practical considerations for safely rolling out new versions in production environments.
The presentation offers a comprehensive overview of online experiment fundamentals, design variations, and real-world case studies from Tencent Lookpoint, emphasizing hypothesis validation, causal analysis, best practices, and actionable recommendations for improving product growth and decision‑making.
This article presents a comprehensive overview of online experiment design and analysis, covering basic definitions, AB testing principles, complex experiment types, real-world case studies from Tencent's information flow platform, and practical guidelines for reliable experiment evaluation and product decision‑making.
This article explains the concept of progressive delivery, its industrial origins, how cloud‑native tools like Kubernetes, Traefik and Spinnaker enable automated A/B testing, canary and gray releases, and the practical benefits for developers, operations, and product teams when releasing features to millions of users.
This report details a series of over twenty A/B tests conducted across multiple product divisions, revealing how targeted design tweaks—such as contrasting guide buttons, fresh promotional visuals, prominent coupon styling, focused product ads, and interactive floor modules—significantly improve user click‑through rates and conversion.
This article outlines a comprehensive A/B testing workflow, covering historical issues, business test process improvements, detailed implementation steps, SQL validation scripts, data verification in analytics platforms, and practical notes to ensure accurate experiment data collection and analysis.
This article explains why A/B testing is essential, illustrates real-world examples from ByteDance, details the multi-layer architecture of the Volcano Engine A/B testing system, outlines experiment design, implementation, statistical analysis, best practices, and future trends, providing a comprehensive guide for product teams.
This article explains the architecture and key components of Toutiao’s recommendation system, covering system overview, content analysis, user tagging, evaluation methods, and content safety measures, and discusses practical implementation details such as feature engineering, model training, recall strategies, and online experimentation.
This article examines the core challenges of running large‑scale A/B testing platforms—supporting thousands of engineers, generating fast reports from massive data sets, and reducing sampling variance—to enable data‑driven product decisions in the AI era.
This article explains how Kuaishou applies causal inference frameworks, such as Rubin's potential outcomes and Pearl's causal graphs, together with machine‑learning techniques like double‑machine learning, causal forests, and meta‑learners to evaluate product features, recommendation strategies, and user behavior under complex network effects in live streaming.
This case study explores how Baidu’s Lemon Aesthetic brand leveraged Baidu App’s natural traffic to transform search queries into a structured medical‑beauty knowledge experience, using user research, KPI‑driven design, MVP validation, and A/B testing to boost engagement, conversion, and brand authority.
This article explains why A/B testing is essential, defines the methodology, details Volcano Engine's multi‑layer A/B testing architecture, outlines client and server experiment flows, shares statistical analysis practices, best‑practice guidelines, future trends, and answers common questions.
This article presents Dr. Jin Yaran’s comprehensive overview of causal inference challenges, frameworks, and practical case studies—including DID, double machine learning, causal forests, and meta‑learners—applied to Kuaishou’s live‑streaming product, and discusses complex experimental designs such as bilateral and time‑slice experiments.
This article explains the statistical principles behind A/B testing, covering concepts such as populations, samples, parameters, hypothesis testing, significance levels, t‑tests, metric types, p‑value calculations, and real‑world examples to guide data‑driven product decisions.
This article describes the evolution of a multi‑objective deep‑learning framework for automatically assigning CRM opportunities to salespeople, detailing five model versions—from an XGBoost baseline with sample weighting to advanced PLE‑based architectures—while reporting offline and online performance gains in both call‑out and connection‑out conversion rates.
The paper presents a comprehensive A/B testing framework for online experiments that guides practitioners through four stages—designing objectives and metrics, implementing random traffic allocation with robustness checks, evaluating effects using descriptive statistics and hypothesis testing, and making rollout decisions based on multidimensional significance and attribution analyses.
The article explains the concept of gray release, outlines a basic architecture with strategy configuration, execution, and service registry components, describes common traffic-splitting strategies, and details practical implementations using Nginx, gateway services, and complex scenarios involving data synchronization and message queues.
This article details the design and multi‑stage optimization of an A/B testing metric system, describing its product architecture, Spark‑based computation engine, ClickHouse OLAP layer, cumulative calculation improvements, and batch processing techniques that reduced processing time from hours to a few minutes for hundreds of experiments and metrics.
This article introduces the motivation and logic behind A/B testing, highlights common misunderstandings of p‑values, and demonstrates how Bayesian A/B testing using PyMC3 can provide intuitive probability statements about which variant performs better, complete with Python code examples.
This article compares popular zero‑downtime deployment strategies—including Blue/Green, A/B testing, Canary (gray) release, Rolling updates, and Red‑Black—explaining their principles, advantages, drawbacks, and ideal scenarios for modern cloud‑native and DevOps environments.
This article explains why internet products need a gray release system, outlines its basic architecture, describes common gray release strategies, and shows how to implement them using Nginx, gateway services, and database handling for complex deployment scenarios.
This case study reveals how Liulishuo’s design and product teams leveraged check‑in gamification, friend‑invitation flows, and strategic banner placements—backed by rigorous A/B testing and data‑driven insights—to boost user engagement, optimize information‑flow ads, and improve overall product performance.
In a ByteDance tech open‑day talk, Yang Zhenyuan explains how clear, measurable goals, the limits of user‑profile and usage‑time metrics, and careful A/B test design are essential for data‑driven product decisions, while highlighting practical pitfalls and real‑world examples.
This article explains the concept of gray (canary) release, outlines a basic architecture with essential components, describes common gray release strategies such as header, cookie, and parameter based routing, and provides practical guidance for implementing gray releases using Nginx, gateway services, and handling complex scenarios like multi‑service and database migrations.
This article explains gray release (canary deployment) as an A/B testing‑style rollout, outlines a simple architecture with strategy configuration, execution, and service registry components, describes common traffic‑splitting strategies, and discusses implementation details for Nginx, gateway layers, and complex multi‑service and data‑centric scenarios.
This guide explains how to use Ingress‑Nginx 0.21+ Canary annotations to perform blue‑green, canary, and A/B testing deployments on Kubernetes, covering annotation syntax, priority rules, and step‑by‑step examples with weight‑based, header‑based, and cookie‑based traffic splitting.
This article presents a graph‑neural‑network approach to predict user influence, cluster users with distributed Louvain methods, and conduct network‑aware A/B experiments that accurately evaluate large‑scale marketing campaigns despite strong network effects.
Tubi’s Popper platform combines a Scala‑based experiment engine, reproducible JSON‑stored configurations, a React UI, and data pipelines using Spark and Akka to enable fast, cross‑team A/B testing, automated analysis, health checks, and data‑driven decision making across mobile and OTT services.
At Tubi, the Popper experiment engine—a Scala‑based, Akka‑powered backend service—combined with a self‑serve UI, automated analysis pipelines, and rigorous health checks, enables teams across ML, mobile, and OTT to run scalable A/B tests, rapidly iterate, and make data‑driven product decisions.
This article explains how to implement blue‑green and canary releases on Kubernetes using Ingress‑Nginx’s annotation‑based canary feature, covering weight‑based traffic splitting, header‑ and cookie‑driven routing, with step‑by‑step YAML examples and command‑line testing for validation.
This article explains the concept of gray release, outlines a simple architecture with essential components, describes common routing strategies, and shows how to implement gray releases using Nginx, gateway services, and complex multi‑service scenarios to ensure controlled, low‑risk deployments.
This article explains how Alibaba’s CBU team tackled decision overload by building an intelligent UI that uses user‑behavior and product‑preference models, replaces algorithmic cold‑start, reduces reliance on traffic, and delivers over 10% PVctr growth across multiple holiday campaigns through systematic tagging, low‑code material development, and rigorous A/B experimentation.
This case study details how a thorough redesign of 58's commercial purchase page—grounded in user interviews, behavior‑model alignment, clearer discount presentation, real reviews, and A/B testing—significantly increased purchase conversion rates, revenue, and customer satisfaction while informing future product improvements.
This article explains the concept of gray release, outlines a simple architecture with essential components, describes common strategies such as header, cookie, and parameter based routing, and provides detailed implementation guidance for Nginx, gateway, and complex multi‑service scenarios.
The article details OkCupid’s year‑and‑a‑half migration from a REST API to a production GraphQL API, describing the four‑step process—page selection, schema construction, shadow requests, and A/B testing—along with lessons learned about error handling, business‑logic placement, and performance monitoring.
This case study examines how a Chinese local services platform tackled information asymmetry between providers and users by replacing phone calls with an online appointment system, introducing a prepaid reservation fee, redesigning UI to reduce friction, and using A/B testing to achieve significant conversion improvements.
This article explains what A/B testing is, how designers can use it to resolve conflicting design choices, outlines a step‑by‑step workflow with real‑world examples, discusses its limitations, and shares key findings that improve product experience and conversion rates.
This article explores why and when to refactor legacy projects, outlines common pitfalls such as missing functionality and inadequate testing, and offers practical guidance on assessment, testing, branching, feature toggles, and A/B testing to ensure a successful rewrite.
This article shares Tubi's practical experience in rapidly iterating machine‑learning systems, emphasizing the early importance of simple end‑to‑end A/B testing platforms, clear launch plans, heat‑based and embedding‑based ranking models, and a culture of fast experimentation over complex deep‑learning research.
The article shares Tubi's practical experience in building a fast‑iterating machine‑learning platform, emphasizing early measurement, simple end‑to‑end A/B testing, clear launch plans, lightweight popularity and embedding models, and rapid experimentation to drive product decisions.
