This article explains how to implement blue‑green and canary releases on Kubernetes using Ingress‑Nginx’s annotation‑based canary feature, covering weight‑based traffic splitting, header‑ and cookie‑driven routing, with step‑by‑step YAML examples and command‑line testing for validation.
This article explains the concept of gray release, outlines a simple architecture with essential components, describes common routing strategies, and shows how to implement gray releases using Nginx, gateway services, and complex multi‑service scenarios to ensure controlled, low‑risk deployments.
This article explains how Alibaba’s CBU team tackled decision overload by building an intelligent UI that uses user‑behavior and product‑preference models, replaces algorithmic cold‑start, reduces reliance on traffic, and delivers over 10% PVctr growth across multiple holiday campaigns through systematic tagging, low‑code material development, and rigorous A/B experimentation.
This case study details how a thorough redesign of 58's commercial purchase page—grounded in user interviews, behavior‑model alignment, clearer discount presentation, real reviews, and A/B testing—significantly increased purchase conversion rates, revenue, and customer satisfaction while informing future product improvements.
This article explains the concept of gray release, outlines a simple architecture with essential components, describes common strategies such as header, cookie, and parameter based routing, and provides detailed implementation guidance for Nginx, gateway, and complex multi‑service scenarios.
The article details OkCupid’s year‑and‑a‑half migration from a REST API to a production GraphQL API, describing the four‑step process—page selection, schema construction, shadow requests, and A/B testing—along with lessons learned about error handling, business‑logic placement, and performance monitoring.
This case study examines how a Chinese local services platform tackled information asymmetry between providers and users by replacing phone calls with an online appointment system, introducing a prepaid reservation fee, redesigning UI to reduce friction, and using A/B testing to achieve significant conversion improvements.
This article explains what A/B testing is, how designers can use it to resolve conflicting design choices, outlines a step‑by‑step workflow with real‑world examples, discusses its limitations, and shares key findings that improve product experience and conversion rates.
This article explores why and when to refactor legacy projects, outlines common pitfalls such as missing functionality and inadequate testing, and offers practical guidance on assessment, testing, branching, feature toggles, and A/B testing to ensure a successful rewrite.
This article shares Tubi's practical experience in rapidly iterating machine‑learning systems, emphasizing the early importance of simple end‑to‑end A/B testing platforms, clear launch plans, heat‑based and embedding‑based ranking models, and a culture of fast experimentation over complex deep‑learning research.
The article shares Tubi's practical experience in building a fast‑iterating machine‑learning platform, emphasizing early measurement, simple end‑to‑end A/B testing, clear launch plans, lightweight popularity and embedding models, and rapid experimentation to drive product decisions.
This article details how 58.com leverages deep learning models such as DNN, Wide&Deep, DeepFM, DIN and DIEN, combined with extensive user‑behavior feature engineering, offline vectorization, and online TensorFlow‑Serving pipelines to improve home‑page recommendation click‑through rates and overall platform efficiency.
This article explains why designers need to understand data, outlines key quantitative and qualitative metrics, and demonstrates how data-driven decisions improved an emoji‑search feature, while emphasizing that intuition remains essential alongside analytics.
This article introduces uplift (incremental) modeling as a causal inference technique for intelligent marketing, explains its mathematical formulation, compares response and uplift models, describes various modeling approaches such as two‑model, one‑model, and label‑transformation methods, outlines evaluation metrics like Qini and AUUC, and demonstrates practical deployment in a real‑world real‑estate platform.
This article provides a comprehensive technical overview of 今日头条's recommendation system, covering its three-dimensional feature model, algorithm choices, real‑time training pipeline, recall strategies, content analysis, user tagging, evaluation methods, and content‑safety mechanisms.
Four A/B experiments conducted for a 618 sales event reveal that overall horizontal scrolling, coupon‑based benefits, top‑left tags, and square slots each outperform their alternatives in click‑through rates and order generation, offering actionable design guidance for e‑commerce platforms.
Youzan created an internal A/B testing platform—combining Java/Node SDKs, a real‑time data pipeline, and a metadata‑driven workflow—to enable data‑driven product iteration, granular traffic allocation, automated logging, statistical analysis, and scalable growth insights across its merchant services, while planning further automation and integration.
This article describes how Meituan's delivery team built a comprehensive A/B testing evaluation platform, covering the motivation for a robust assessment framework, the architecture of the platform with three functional modules, the statistical methods for reliable experiment design, and the practical implementation details that enable data‑driven operational decisions.
The article details Meituan Delivery’s A/B testing platform and evaluation system, explaining its closed‑loop design, multi‑strategy traffic allocation with AA grouping, comprehensive metric hierarchy, statistical rigor, data integration, and implementation architecture, and outlines future tools for traffic‑volume recommendation.
This article explains the design of a standardized, simple, and efficient front‑end A/B testing pipeline, covering experiment configuration, data models, platform architecture, runtime JSSDK, traffic‑splitting strategies, and data back‑flow to enable reliable, data‑driven product decisions.
This article presents a series of A/B experiments comparing flat expansion versus collapse, paging swipe, and horizontal swipe layouts, revealing how different UI designs affect revenue, user value, click-through, and conversion rates in e‑commerce venues.
This article explains the principles, advantages, and drawbacks of blue‑green deployment, canary (gray) release, and A/B testing, compares their use cases, and provides step‑by‑step guidance for safely rolling out new versions in modern cloud‑native environments.
This article explores how large wealth‑management platforms can model product recommendation as a mapping between customers and financial products, defines various evaluation goals such as transaction volume, revenue and user satisfaction, and outlines a systematic A/B‑testing workflow for comparing and optimizing recommendation algorithms.
This article explains the core modules of an A/B testing system, details a step‑by‑step design workflow using an internet‑finance example, and highlights key design principles such as scientific traffic allocation, sufficient data, rigorous statistical analysis, and continuous iteration for data‑driven product optimization.
This article explains the design and implementation of an ABT (Auto‑Bidding Test) system that automates A/B decision cycles, covering client SDK architecture, protocol specifications, data‑point collection, resource‑placement experiments, and home‑page integration to accelerate product growth.
This article explains the challenges of validating test cases in production environments, especially for cross‑browser scenarios, and outlines practical strategies—including Selenium Grid automation, blue‑green, canary, and A/B testing—to ensure reliable, efficient production testing.
The talk explains uplift modeling for intelligent marketing, showing how causal lift predictions—derived from randomized experiments using two‑model, one‑model, or tree‑based methods—identify truly responsive users, evaluate performance with AUUC/Qini, and were applied to Taopiaopiao’s coupon allocation via knapsack optimization, highlighting challenges and future directions.
This article provides a comprehensive technical overview of Toutiao’s recommendation system, covering its three‑dimensional modeling approach, feature engineering, user‑tag pipelines, real‑time training infrastructure, evaluation methodology, and content‑safety mechanisms.
This article provides a comprehensive overview of Toutiao's recommendation system, covering its three-dimensional modeling of content, user, and environment features, various algorithmic approaches, feature extraction, real‑time training pipelines, recall strategies, user‑tag engineering, evaluation methods, and content‑safety measures.
The article describes how Meituan Delivery engineered a unified, end‑to‑end machine learning platform—named Turing—to streamline data processing, feature engineering, model training, deployment, online prediction, and A/B testing, thereby improving algorithm iteration speed, scalability, and operational efficiency for its massive real‑time delivery service.
Feature flags enable controlled system changes, allowing teams to observe business and technical impacts, retain beneficial updates, quickly roll back harmful ones, and continuously learn through a fast feedback loop that guides subsequent modifications.
This article explains the principles of A/B testing, the statistical concepts such as population, sample, hypothesis testing, p‑values and t‑tests, describes how to calculate metrics for rate and mean indicators, and illustrates a real‑world experiment with detailed evaluation methods.
This guide explains what A/B testing is, when it should be used, outlines an eight‑step framework—from building product funnels to analyzing results—and discusses specific challenges and solutions for applying A/B testing in e‑commerce environments.
This guide explains how to keep a feature flag's assignment consistent for individual users as exposure percentages increase or decrease, using a new "Previously Seen Products" section on an e‑commerce search page as a practical example.
This article explains the challenges of low‑frequency home‑page UI A/B testing in a mobile app and presents a hybrid client‑server solution that assigns UUIDs to users, uses MurmurHash for logical grouping, maps to business groups, and outlines server‑side handling and best practices.
This article explains how Alibaba’s Hand‑Taobao platform uses a full‑link intelligent delivery framework—combining user intent recognition, rights recommendation, and advanced machine‑learning models such as XFTRL and Thompson Sampling—to predict activity drops during the Spring Festival and deliver personalized interventions that significantly improve DAU, click‑through, and redemption rates.
This article presents a comprehensive overview of the evolution of Weibo's advertising strategy engineering architecture, detailing the system's growth from early banner ads to a sophisticated, multi‑layered online advertising platform that integrates algorithmic models, A/B experimentation, real‑time data pipelines, and precision targeting to support scalable, high‑performance ad delivery.
This article details Kuaikan Comics' personalized recommendation pipeline, covering business context, diverse content formats, technical challenges, content‑based and collaborative‑filtering methods, ranking models, system architecture, A/B testing, and future directions for improving recommendation quality.
This article provides a comprehensive introduction to recommendation systems, covering their definition, overall offline and online architectures, feature engineering, collaborative filtering, latent semantic models, ranking algorithms, and evaluation methods including A/B testing and offline metrics.
This article shares extensive practical experience on recommendation system optimization, outlining the importance of problem definition, the limits of AB testing, and four guiding principles—avoid fundamentally wrong actions, do the right things correctly, keep solutions simple, and prevent over‑optimization.
This article explains how A/B testing serves as a vital measurement and optimization tool for internet products, covering metric definition, experiment management platforms, traffic splitting strategies, orthogonal and exclusive rules, and essential statistical concepts such as hypothesis testing, t‑test, z‑test, and p‑value analysis.
The paper proposes a video recommendation framework that combines recall and ranking modules, using a multi‑modal topic clustering approach—integrating audio, visual, and textual features via NeXtVLAD, PCA, and K‑Means—to generate unified video representations, improve candidate selection, and boost click‑through and viewing time, while addressing cold‑start and semantic relevance challenges.
The NVID recommendation system for Taobao is built on a four‑layer architecture—activity material, configuration, business process, and application—and solves environment isolation, performance, audience management, and A/B testing challenges through optimized data schemas, ID mapping, multi‑level caching with database fallback, and real‑time user targeting, while future work aims at personalized audiences and automated ad optimization.
This article offers a comprehensive, system‑engineer‑focused guide to Learning to Rank, covering fundamental machine‑learning concepts, evaluation metrics, training approaches, and a detailed online ranking architecture with feature, recall, and model governance, illustrated by real‑world examples from Meituan‑Dianping.
This article provides a comprehensive, system‑engineer‑focused guide to Learning to Rank, covering fundamental machine‑learning concepts, evaluation metrics such as Precision, nDCG and ERR, training‑testing‑inference stages, pointwise/pairwise/listwise methods, and a detailed multi‑layer online ranking architecture with feature, model and recall governance.
This article explains how to use feature toggles in .NET Core to hide or gradually release functionality, covering built‑in toggle types, open‑source libraries, configuration via appsettings, custom toggle creation, and step‑by‑step code examples for practical implementation.
This article explains the fundamentals of recommendation algorithms, illustrates their modest impact with real‑world examples, and outlines how modern e‑commerce systems collect data, rank items, and use rapid A/B testing to continuously improve personalized recommendations.
The Sun Dial platform is a universal A/B testing system built for 58.com that supports single‑layer and multi‑layer experiments, provides uniform traffic splitting, real‑time OLAP analytics with Druid, and offers a web interface for easy configuration, enabling data‑driven product optimization across multiple business lines.
The article describes the motivation, layered design, traffic allocation, statistical validation methods, and system architecture of a scalable A/B testing platform that enables multiple concurrent experiments while ensuring independent traffic segmentation and reliable data analysis for product growth.
This article outlines the challenges of algorithm testing in real‑world engineering, proposes a step‑by‑step testing framework—from understanding business context and verifying data exchanges to evaluating performance metrics and iterating improvements—while offering practical advice and examples.
This article explains the fundamentals of A/B testing, illustrates its evolutionary analogy with Darwin's finches, highlights typical mistakes such as limited variants and improper sample division, and provides a step‑by‑step guide for designing rigorous experiments to drive product growth.
Learn how A/B testing and multivariate testing can identify the most effective UI designs, improve user experience, and increase conversion rates, while exploring essential tools such as Google Optimize, Crazy Egg, and Optimizely, and best practices for planning, executing, and analyzing experiments.
This article explains the statistical foundations of A/B experiments—including population, sample, sampling error, confidence intervals, hypothesis testing, type I/II errors, statistical significance, and power—so engineers can design reliable tests and interpret results with confidence.
This article introduces Spring's BeanPostProcessor, explains its two callback methods, and demonstrates through a real‑world A/B testing scenario how custom annotations and a post‑processor can inject dynamic proxies to simplify routing logic and improve code maintainability.
The article explains hypothesis testing in A/B experiments, describing null and alternative hypotheses, type I and II errors, p‑values, statistical significance versus practical impact, confidence intervals, statistical power, sample‑size planning, and a checklist for interpreting results responsibly.
This article explains the essential statistical concepts—populations, samples, sampling error, confidence intervals, the Central Limit Theorem, and normal distribution—that underpin A/B testing, showing how they enable reliable hypothesis evaluation, accurate impact prediction, and data‑driven decision making for product experiments.
This article examines Toutiao’s rapid rise as a personalized news platform, detailing its AI‑driven recommendation pipeline, web‑crawling infrastructure, similarity‑matrix algorithms, A/B testing, and the role of human moderation in delivering highly targeted content to billions of users.
Didi’s Apollo platform combines A/B testing with gradual (gray) release, letting product teams safely roll out new features to targeted user segments, monitor key metrics, and apply best‑practice guidelines—such as isolating variables, pre‑defining metrics, controlling duration, random grouping, and confidence analysis—to achieve statistically significant, data‑driven improvements across thousands of weekly releases.
Facebook evaluates its Newsfeed recommendation quality through three pillars—machine-learning model metrics like AUC, extensive product data KPIs such as DAU and interaction rates, and user-survey feedback—while maintaining long-term backtests and emphasizing the risks of relying on a single metric.
The article describes Dropbox’s Stormcrow system, a configurable feature‑gate platform that enables fast, safe rollout of new functionality across web, desktop, and mobile clients, supports granular A/B testing, leverages custom data fields, and integrates deployment, monitoring, and audit tooling for large‑scale operations.
This article explains a data‑and‑experience‑driven framework—Find, Split, Test—for assessing design effectiveness and iterating a group‑buying product, illustrating each step with concrete metrics, flow analysis, A/B experiments, and measurable conversion improvements.
This article examines the concept, benefits, and practical implementation of A/B testing in iOS development, covering design considerations, code organization patterns such as selector caching, strategy pattern, and protocol dispatchers, as well as related build‑time concerns like static‑library merging and CocoaPods subspecs.
This article explores how Ctrip integrates data analysis with product design, presenting two detailed case studies— the evolution of the homestay channel and the optimization of the Guesthouse app order detail page— to illustrate the role of data, user research, and A/B testing in driving user‑centered product improvements.
This article explains the basic principles, practical demo, platform architecture, statistical validation, sample size estimation, and reporting workflow of A/B testing used at Qunar to evaluate advertising strategies and product features, illustrating how data‑driven experiments are designed, executed, and analyzed.
This article examines Toutiao’s product evolution—from strategic naming decisions validated by A/B testing to its data‑driven recommendation architecture—highlighting how lean product methodology, algorithmic personalization, and continuous experimentation underpin its success in the mobile news landscape.
The article presents a comprehensive overview of mobile growth methodology, covering the definition of growth, its rising importance in China, core AARRR metrics, experimental workflows, cross‑functional team structure, mobile‑specific challenges, user onboarding funnels, A/B testing architecture, deep‑linking techniques, and practical recommendations for incremental, cost‑effective implementation.
This article explains how a simple 20‑line multi‑armed bandit implementation can replace traditional A/B testing by continuously balancing exploration and exploitation to automatically discover the most effective UI variant, reducing manual analysis and improving conversion rates.
This article explains how leveraging CDN features such as redirects, A/B testing, URL rewriting, high‑concurrency user management, access control, and form input control can streamline web operations, improve reliability, and accelerate time‑to‑market for modern online businesses.
This article explains how the AirTrack platform combines A/B testing, dynamic experiment conditions, a real‑time SDK, and a data feedback system to let mobile teams quickly validate decisions, perform gray releases, and personalize features without waiting for full app version cycles.
This article describes Meituan's online search ranking architecture for O2O services, covering data pipelines, feature loading, ranking service workflow, A/B testing, model choices, cold‑start handling, and position bias mitigation, all tailored for mobile‑centric personalized ranking.
This article examines the trade‑off between tab navigation and side‑drawer navigation in a multi‑page mobile app, sharing A/B test results that show a dramatic drop in user engagement when the drawer was introduced and offering practical guidance on when each pattern is appropriate.
