The paper proposes CAIM, a context-aware user intent modeling framework for e-commerce relevant recommendation that decomposes intent into category, brand, shop, and price sub-intents via three modules (IEM, IRM, SIEM), achieving 0.20–0.21 AUC gains offline and 6.9% order and 7.7% GMV lifts in a 14-day online A/B test.
Kuaishou's KwaiMind unifies general and e-commerce image editing via a four-agent data engine and DiffusionOPD multi-teacher distillation, achieving top scores on public benchmarks and the custom Ecom-Bench, with a 2.44% relative CTR increase in online A/B tests.
Airbnb engineers detail how they adapted search ranking for map interfaces by modeling user attention through three iterations — uniform distribution, tiered markers, and decaying attention from map center — validated via A/B tests that increased bookings by 0.27% and reduced map interactions by 1.5%.
Adding three secondary indexes to a MySQL 8.0 orders table caused write P99 latency to jump from 15ms to 2.3s and TPS to drop from 4800 to 1100 during peak hours; the article details a reproducible forensic process using A/B testing, invisible indexes, and sysbench to prove the culprit index had zero read benefit and safely remove it.
This article details the multi-path recall architecture in recommender systems, covering five recall sources (tag, vector, popularity, collaborative filtering, exploration), the Recall Manager's parallel scheduling and circuit breaking, engineering deployment pipelines with offline/online evaluation, and future trends like generative recall and joint optimization.
This article explains why traditional A/B testing fails in AI‑driven products, then details hierarchical traffic‑control, data‑isolation architectures, evaluation metrics, golden‑dataset construction, and end‑to‑end multi‑agent testing, providing concrete code, industry examples, and step‑by‑step guidelines for reliable AI application assessment.
In 2026, AI‑powered A/B testing platforms compress the experiment lifecycle from weeks to minutes by automating hypothesis generation, SDK deployment, federated data fusion, and Bayesian stopping, while also introducing multi‑layer statistical safeguards and organization‑wide Experiment‑as‑a‑Service, fundamentally reshaping product experimentation compared to traditional manual workflows.
Google is experimenting with a new homepage for unsigned users that replaces the traditional Search button with AI‑driven shortcuts like Create Image, Ask File, and Brainstorm, some of which work without signing in, a change observed in Chrome, Edge, and Perplexity’s Comet browser.
The article explains how moving feature switches from hard‑coded if‑else statements to a platform‑based runtime system enables granular targeting, gradual rollouts, A/B experiments, instant kill‑switches, and SDK‑based local evaluation that can handle millions of queries per second without redeployments.
The article explains how a tiny tweak in a system prompt can trigger a three‑day outage, then details the L3 context layer that organizes prompts, version‑controls them, allocates token budgets, compresses context, runs A/B tests, and compares open‑source and SaaS PromptOps platforms for reliable LLM deployments.
AgentX introduces an agent‑driven closed‑loop that automates idea generation, code production, online A/B testing, and experience consolidation, boosting experiment concurrency eight‑fold, increasing per‑person business value 3.7×, and delivering over 0.5% app‑time growth and more than 1 billion RMB annual revenue.
Kuaishou’s new Recommendation-as-Generation (RaG) framework replaces traditional retrieve-and-rank with a generative pipeline that predicts user interests, creates personalized video content, and closes the loop with feedback, delivering a 1.87% ad‑revenue lift for over 400 million daily users.
The article details a data‑driven redesign of a mini‑game home page that shifted focus from increasing click volume to extending user playtime, removed childish visual cues, introduced a youthful and relaxed brand language, and leveraged A/B testing and layered visual hierarchy to improve engagement and brand perception.
The article explains why prompts, now treated as a measurable software interface, become a performance bottleneck in AI-native apps, and presents a four‑quadrant methodology—including observability, quantification, attribution, and governance—plus five concrete optimization tactics backed by real‑world case studies.
In 2026, a leading e‑commerce platform cut A/B test preparation from 3.2 days to 47 minutes, boosted experiment throughput 4.3× and GMV 2.8% by applying a three‑step automation workflow, an intelligent analysis engine with three‑level reasoning, and a three‑tier human‑machine decision guard.
In 2026, AI‑driven A/B testing platforms became standard, cutting experiment cycles by 63% but raising false‑positive rates to 19.4%, and this article reveals five critical mistakes—from mistaking auto‑traffic split for true randomization to ignoring metric drift and business impact—that can undermine results.
The article examines how 2026’s new A/B testing automation paradigm—combining dynamic traffic allocation, real‑time causal modeling, metric‑autonomy systems, and built‑in privacy compliance—dramatically shortens experiment cycles, boosts statistical power, and transforms experimentation from a manual chore into a scalable, trustworthy decision engine.
The article details a mid‑size SaaS growth team’s transformation from manual, experience‑driven A/B testing to a fully automated, auditable end‑to‑end decision flow, describing the pitfalls of pseudo‑automation, a three‑layer automation engine, and cultural shifts that boosted experiment adoption from 41% to 89% and cut decision latency from 3.7 days to 4.2 hours.
This guide lists six open‑source AI skills that help product managers turn vague ideas into concrete designs, write structured PRDs, break requirements into executable plans, set up rigorous A/B experiments, implement analytics tracking, and optimize new‑user onboarding, each with install commands and usage examples.
Anthropic's March 2026 skill‑creator update adds five engineering‑focused functions—Evals, Benchmark, multi‑agent parallelism, A/B testing, and trigger optimization—enabling systematic testing, performance tracking, and a reported 83.3% improvement in trigger success across public skills.
The article examines four common yet hidden model evaluation mistakes—confusing attractive metrics with business impact, using static test sets, ignoring statistical significance, and lacking fine‑grained attribution—illustrating each with real‑world cases and offering concrete practices to build a more robust, business‑aligned evaluation pipeline.
This article details how a production‑grade prompt system powered by large language models was designed to replace manual A/B experiment inspection, introducing a six‑level priority decision tree, robust data preprocessing, and systematic bad‑case analysis that lifted automation accuracy from 68% to over 80% while providing clear, explainable recommendations.
This article provides a comprehensive guide to gray releases and A/B testing, covering common scenarios, implementation methods, layered experiment design, hash-based bucket allocation, data collection workflows, statistical analysis, and practical Java and SQL code examples for reliable feature validation.
This article explains the fundamental differences between gray release and A/B testing, provides step‑by‑step guidance for implementing both strategies with Spring Cloud Gateway, Nacos and Kubernetes, and compares container‑level canary deployments with gateway‑level traffic routing to help you choose the right approach for reliable production releases.
The DMF framework introduces a modality‑enriched Decoupled Target Attention (DTA) and a Complementary Modality Modeling (CMM) strategy that together close the semantic gap between ID and multimodal features, delivering up to 5.3% CTCVR lift, 7.43% GMV increase, and a three‑fold throughput gain in large‑scale e‑commerce recommendation scenarios.
This article explains the core concepts of A/B testing, including significance, p‑values, minimum sample size, experiment duration, common interpretation pitfalls, and practical e‑commerce conversion tips, helping designers and product teams make data‑driven decisions without falling into statistical traps.
The paper reveals that knowledge‑sharing across recommendation domains causes homogenized results that boost per‑scene accuracy yet shrink overall GMV, and proposes the DPOMR algorithm with manual rules and an industrial‑grade reference model to restore domain diversity and improve platform‑wide order volume.
The article examines how conventional single‑sided A/B testing breaks down in two‑sided markets due to SUTVA violations, cross‑interference, and spillover effects, and presents practical mitigation strategies such as small‑world partitioning, counterfactual interleaving, and model‑based corrections.
Google’s Gemini 3.0, spotted through an A/B test on AI Studio, showcases dramatic improvements in coding precision, SVG generation, and multimodal understanding, offering developers faster UI/UX code, larger output lengths, and higher quality than Gemini 2.5, while community discussions highlight its potential and access challenges.
In a competitive hiring market, the team used AI to redesign multi‑platform job cards, addressing three major pain points—homogeneous information, outdated templates, and misalignment with young users—by creating four visual styles and a standardized generation pipeline, resulting in higher click‑through and application rates.
This article details JD Retail's AI‑driven virtual try‑on system, covering business challenges, technical hurdles, core algorithmic innovations, practical results from a major fashion brand, and future directions for personalized, automated, and predictive AI try‑on in e‑commerce.
This article details how JD.com’s AI‑driven virtual fitting solution, integrated with an A/B testing platform, transformed fashion e‑commerce by generating realistic model images and videos, cutting production costs to zero, accelerating design cycles, and increasing conversion rates by over 80% during major sales events.
In an age of information overload, this guide explains visual scanning patterns like F‑ and Z‑reading, offers practical layout, typography, and visual‑guidance techniques, and shows how to validate designs with A/B testing, heatmaps, and performance optimization to make content effortlessly readable.
This article presents a hands‑on guide to using OpenResty—Lua‑enhanced Nginx—for implementing static and dynamic blacklists, fine‑grained rate limiting, A/B testing via upstream selection, and real‑time service quality monitoring, all with production‑ready code examples.
JD Retail’s “JingDianDian” AI virtual try‑on platform leverages a 12‑billion‑parameter Flux‑Fill diffusion model and multimodal pose estimation to automatically create realistic model images and videos, integrates with the JingMai A/B testing system, and delivers up to an 80% boost in conversion while cutting production costs and time dramatically.
This article details JD Retail Technology’s AI‑driven virtual try‑on system that combines a 12B Flux‑Fill diffusion model with a high‑quality virtual model library and integrates with the JingMai A/B testing platform, cutting production costs to zero, slashing cycle time to half a day, and increasing order conversion rates by over 80% during the 618 shopping festival.
This article details how JD.com’s AI-powered virtual try‑on system, integrated with the Jingmai A/B testing platform, transformed fashion e‑commerce by generating realistic model images and videos, reducing production costs to near zero, cutting design cycles from weeks to hours, and increasing conversion rates by over 80% during major sales events.
This article explores why outliers appear in large‑scale internet A/B tests, explains their impact on experiment precision, compares traditional trim and winsorize techniques, reviews a range of statistical and machine‑learning detection methods, and offers practical recommendations for handling them in product experiments.
This article explores why outliers destabilize online A/B tests, explains their statistical definitions, compares trimming and winsorizing techniques, reviews classic and machine‑learning detection methods, and offers practical guidance for applying these approaches to improve experiment reliability.
This comprehensive guide explores the fundamentals of randomized controlled experiments, discusses classic RCT designs and their limitations, and presents advanced methods such as CUPED variance reduction, stratified, paired, and covariate‑adaptive randomization, as well as spill‑over modeling and random saturation designs to improve experimental power and reliability.
This whitepaper introduces the importance of data‑driven A/B testing, outlines its theoretical foundations, practical challenges such as small samples and spillover effects, and presents a structured roadmap—including experiment basics, statistical principles, advanced designs, and SDK implementation—to help practitioners design trustworthy experiments.
In today's fast‑changing market, traditional intuition‑based decisions falter, but systematic A/B testing—illustrated by ByteDance’s academic‑loop culture and real‑world case studies—empowers organizations to replace guesswork with evidence, accelerate innovation, and achieve measurable performance gains across products and strategies.
This article explains the LIFT model—a six‑factor framework for landing‑page conversion optimization—detailing its components, a four‑stage workflow from problem identification to result analysis, and practical tips for applying the model to everyday design projects.
This article shows how a product manager transformed intricate promotion rules—like "spend 300, get 50 off" with exclusions—into a visual, low‑code builder using modular components, natural‑language parsing, safety checks, version control, A/B testing, and collaborative approval flows.
This article examines the hidden complexities of A/B experiments in short‑video recommendation feeds, explains why traditional designs produce biased results due to learning, double‑sided, and network effects, and presents practical double‑sided and community‑randomized experiment frameworks to obtain unbiased strategy evaluations.
This article examines the challenges of outliers in large‑scale internet A/B testing, explains their statistical definition, outlines common causes, evaluates the benefits and limits of removal, and compares traditional trim and winsorize techniques along with practical detection and risk‑control strategies.
The article outlines how Minsheng Bank designed a closed‑loop A/B testing system—covering culture, platform tools, and experiment mechanisms—to support data‑driven decision making during its digital transformation, detailing challenges, implementation methods, real‑world case studies, and future automation trends.
The article explains why outliers destabilize internet A/B tests, outlines their causes, compares trimming and winsorizing, presents lightweight detection (e.g., kurtosis) and risk‑control strategies, and offers practical recommendations for bias‑aware removal and variance‑reduction techniques to improve experimental precision.
The Beta distribution is a flexible probability model defined on the interval [0,1] with two shape parameters that control its form, offering useful properties such as mean and variance, and is widely applied in A/B testing, risk assessment, and machine‑learning tasks to model proportions and uncertainties.
This article provides a comprehensive analysis of the HLLM paper, detailing the motivation behind using large language models for recommendation, the hierarchical architecture of Item and User LLMs, the training objectives, extensive offline and online experiments, scaling behavior, and practical deployment insights.
The article presents a next‑generation experiment analysis engine that standardizes the core AB testing framework, integrates advanced statistical solutions to tackle small‑sample and overflow challenges, and offers precise variance and P‑value calculations, thereby improving reliability and efficiency for multi‑side fulfillment platform experiments.
This article shares a data‑centric perspective on user growth, covering entropy reduction, information management, the data‑driven flywheel, A/B testing practices, retention strategies, and practical case studies that illustrate how systematic data analysis fuels sustainable product expansion.
The article reviews the development of online A/B testing, covering sampling and traffic‑splitting techniques, metric computation improvements, statistical inference advances, and current challenges such as interference, real‑time inference, and large‑scale metric computation, while referencing recent research papers.
This article explains the fundamentals of A/B testing, outlines its key benefits such as data‑driven decisions, improved user experience and higher conversion rates, and provides practical best‑practice guidelines, essential tools, and required skills for successful implementation.
This article introduces causal inference fundamentals, distinguishes correlation from causation, reviews major methodological streams, and demonstrates how uplift and gain models—implemented with T‑learner, S‑learner, and tree‑based approaches—can be applied to user growth and marketing scenarios, including evaluation metrics and future challenges.
This article explains how A/B testing and model grayscale are applied in credit risk control, discusses the specific requirements for effective testing, compares upstream and risk‑system traffic splitting methods, and proposes an integrated all‑in‑one solution to simplify feature engineering, model evaluation, and deployment.
This talk explains how combining heavy randomization with regression adjustment can effectively mitigate AA problems in A/B testing, improving experiment credibility by addressing covariate imbalance and enhancing result validity for data‑driven decision making.
This article explains how A/B testing and feature‑flag experiments enable faster iteration, safer rollouts, and data‑driven optimization for large‑model AI products, offering practical steps and real‑world scenarios to improve quality and efficiency.
Speaker Wanbo Kui, a Didi data analyst, will present a systematic approach to addressing the AA problem in random experiments, covering academic and industry research on re-randomization, its principles and simulations, practical applications, and how it enhances experiment validity.
This presentation introduces the AA problem that can compromise A/B test validity and explains how combining re‑randomization with regression adjustment provides an effective, practical solution to improve experiment reliability and credibility.
The DataFun Data Science Summit on May 25 will feature Tencent data scientist Li Yilin presenting a comprehensive talk on pairwise‑data A/B experiments, covering unbiased estimation under various randomizations, theoretical analysis, and practical insights for causal inference in network‑driven online experiments.
The DataFun Data Science Summit on May 25 will feature Tencent data scientist Li Yilin presenting a comprehensive overview of paired‑data A/B experiments, covering causal inference challenges, unbiased estimators under various randomization designs, theoretical analysis, and practical insights for network‑based online experiments.
This article reviews the two‑decade evolution of online A/B testing, outlines the business and technical challenges enterprises face, and details three core technical challenges—experiment accuracy, analysis & interpretation, and efficiency—along with practical solutions for each.
DataTester’s A/B testing platform underwent a comprehensive architectural overhaul, applying domain‑driven design, modular refactoring, automated validation, and dependency inversion to reduce change amplification, cognitive load, and unknown unknowns, ultimately improving code readability, maintainability, scalability, and development efficiency across its lifecycle.
Cloud Music revamped its push‑notification system by separating business and channel layers, integrating a unified delivery platform, tailoring messages to Android manufacturers, adding new push channels, refining frequency and copy controls, and using AI‑generated creatives, which together doubled click‑through rates and nearly doubled total click users within two months.
This article explains why and how to conduct A/B testing for UI design, outlines experiment setup, variable creation, and data analysis, and presents detailed case studies of JD Express mini‑program pop‑up and order‑completion page experiments that demonstrate measurable improvements in click‑through and conversion rates.
This article examines why A/B experiments often capture only short‑term impacts, explains external and internal factors behind short‑ and long‑term effects, and presents seven industrial methods—including user‑learning models, personalized recommendation adjustments, surrogate metrics, and bias correction—supported by real‑world case studies.
This article explains search experiments from a data‑product viewpoint, covering AB testing fundamentals, multi‑layer experiment architecture, four experiment types (ordinary AB, vocabulary, diff‑AB, interleaving), real‑world case studies, and a comprehensive FAQ addressing typical challenges and troubleshooting methods.
This article explains the role of A/B experiments in recommendation systems, outlines their workflow, shares practical tips and parameter design strategies, and demonstrates how to use experiment parameters and feature flags for efficient testing, optimization, and full‑scale deployment.
This article explains the principles of A/B testing and the ε‑greedy multi‑armed bandit algorithm, illustrates their practical use in e‑commerce recommendation optimization, and draws broader life lessons about balancing exploration and exploitation for better personal and professional decisions.
In 2022 Tencent's A/B Test team created the overseas SaaS product ABetterChoice, abstracting internal experiment capabilities, adapting to multi‑cloud compliance, and unifying computation with StarRocks, enabling game titles like Honor of Kings, PUBG Mobile, and Ubisoft to run scalable, compliant A/B experiments worldwide.
This article explains how designers can apply statistical methods such as A/B testing, regression, and correlation analysis to identify which subtle product changes truly drive core business metrics like NPS, retention, and conversion rates.
This article explains why variance reduction is essential for A/B experiments, describes at‑assignment and post‑assignment techniques such as stratified sampling, post‑stratification and CUPED, compares their effectiveness, and presents real‑world case studies demonstrating how they improve experiment sensitivity without increasing sample size.
This article examines why A/B experiments often capture only short‑term impacts, categorises external and internal causes of short‑term bias, and presents seven industry‑tested approaches—including user‑learning models, personalized recommendation adjustments, surrogate metrics, and bias correction techniques—to reliably estimate long‑term strategy effectiveness, illustrated with real business cases.
While A/B testing is widely used for data-driven decisions, short-term experimental results often diverge from long-term impacts, leading to misguided strategies; this article examines why such inconsistencies arise and reviews major methods—including extended experiments, holdout groups, post‑analysis, CCD, and surrogate‑metric modeling—to reliably estimate long‑term effects.
This article explains how Huolala's data‑science team tackles interference between multiple time‑slice experiments by using city‑level isolation, nested experiment planning, and bias‑variance trade‑offs, providing detailed guidelines, recovery cycles, and case studies to maximize traffic utilization and experimental reliability.
This article explains how to construct a metric system, identify bottlenecks, and design data‑driven growth strategies using Volcano Engine's DataFinder and DataTester, illustrated with real‑world case studies and step‑by‑step A/B testing practices.
This article explains how to design and implement A/B experiments in complex two‑sided markets, covering core concepts of causality, detailed case studies, various allocation principles, risk‑benefit trade‑offs, and practical guidelines for selecting appropriate experimental methods across different business scenarios.
This article explains ten data‑analysis techniques for enhancing user experience—such as behavior tracking, A/B testing, sentiment analysis, and personalization—and provides concrete SQL code snippets that illustrate how to import, query, filter, sort, aggregate, join, update, delete, and back up data in relational databases.
This article explains the statistical foundations of A/B testing, introduces potential outcomes and average treatment effect, defines metric covariance, and presents practical estimation methods—including naive, data‑augmentation, and bucket‑based approaches—along with real‑world performance evaluations and applications such as variance reduction and Bayesian optimization.
The DataFunSummit2023 online conference brings together experts from Tencent and Kuaishou to present cutting‑edge research on causal inference for large‑scale A/B testing, including deep‑learning‑based multi‑experiment effect estimation, a distributed causal inference framework (Fast‑Causal‑Inference), and strategies for evaluating long‑term policy impacts.
This article explains how Meituan's fulfillment team designs, implements, and operates a reliable A/B testing platform for multi‑side markets, detailing statistical pitfalls, experiment types, traffic-splitting frameworks, and automated analysis pipelines to ensure credible results despite overflow effects, small samples, and fairness constraints.
This article explains how to perform data‑driven A/B testing in progressive delivery using Argo Rollouts Experiments, covering the concepts of progressive delivery, A/B testing fundamentals, the Argo Rollouts architecture, required Kubernetes resources, and step‑by‑step commands and YAML manifests for a weather‑app example.
The presentation explains how to detect, analyze, and leverage regional variations in overseas A/B test results to make more informed product decisions, using a systematic experimental analysis framework grounded in causal inference and online experimentation methods.
The presentation by Tencent PCG data product manager Wang Dongxing introduces A/B testing fundamentals and shares practical experiences with various search experiment methods—including regular A/B, vocabulary, diffAB, and interleaving—while highlighting common pitfalls and offering actionable insights for practitioners.
This article traces the step‑by‑step evolution of an AB testing platform—from its initial 1.0 version with basic traffic splitting, through the 2.0 era that introduced multi‑layer orthogonal traffic models and real‑time metric pipelines, to the 3.0 era focused on usability, stability, and advanced analysis—while sharing concrete design decisions, implementation details, and lessons learned.
This article presents a comprehensive guide to designing, implementing, and analyzing a reliable A/B testing platform for Meituan's multi‑side fulfillment business, covering statistical pitfalls, experiment types, traffic‑splitting frameworks, automated analysis engines, and practical solutions for overflow effects, small samples, and fairness constraints.
This article explains three common software release strategies—blue‑green deployment, rolling deployment, and canary (gray) deployment—detailing their principles, advantages, potential pitfalls, and practical considerations, while also contrasting them with A/B testing and noting related operational concerns.
In an interview with LeKe CTO Chengshi, the company’s rapid growth is attributed to three data‑powered capabilities—fine‑grained user operation, agile A/B‑testing, and AI‑driven personalized recommendation—enabled by Volcano Engine’s data platform and the data‑flywheel concept.
This case study details Baidu’s systematic redesign of its real‑time Q&A service, covering pre‑service question‑page enhancements, in‑service IM page expansion, and post‑service repurchase guidance, illustrating how iterative, data‑driven design lifted user experience and doubled revenue streams.
This article explains why A/B testing is considered the gold standard for causal inference, shares ByteDance’s extensive internal experimentation practices, describes the Volcano Engine platform architecture, outlines how to design and run experiments, and provides real case studies and Q&A for product teams.
This article explains why A/B testing is considered the gold standard for causal inference, shares ByteDance’s extensive internal experimentation practices and case studies, describes the Volcano Engine experiment platform architecture, and outlines the step‑by‑step process for launching reliable A/B experiments.
From its inception as a simple game distribution platform to a sophisticated, multi‑layered recommendation architecture, this article details the background, early models, business growth, architectural evolution, caching strategies, GC optimization, rate limiting, experiment platform, multi‑path recall, dynamic tuning, and future intelligent enhancements of a game recommendation system.
This article explains how Tencent leverages A/B testing across its overseas games, detailing market differences, experimental methodology, multi‑cloud platform compliance, data architecture, and case studies that illustrate how targeted experiments improve user onboarding, gameplay settings, and email‑based re‑engagement.
This article reviews common deployment techniques—blue‑green, rolling, and canary (gray) releases—explaining their workflows, advantages, limitations, and best‑practice precautions to help teams choose the right strategy for reliable production updates.
This article explains how Tencent leverages A/B testing across its overseas games, detailing the current market situation, experimental capabilities, multi‑cloud platform architecture, and case studies that illustrate how data‑driven experiments improve user retention, engagement, and overall business growth.
This article presents a comprehensive guide to growth analysis, covering basic methodologies, construction of a growth metric system, end‑to‑end analysis loops, practical case studies, emerging metric scenarios, and the upgraded professional version of the growth analysis platform.
This article introduces common release strategies—blue‑green deployment, rolling updates, and canary (gray) releases—detailing their workflows, advantages, drawbacks, and practical considerations, and clarifies how they differ from A/B testing in modern software delivery.
This article explains how to perform data‑driven A/B testing in progressive delivery using Argo Rollouts Experiments on Kubernetes, covering the concepts, required resources, YAML manifests, command‑line steps, and the benefits of combining A/B tests with canary deployments.