Intelligent Delivery System for Baidu's Large‑Scale Information Flow Recommendation: Practices and Solutions

Introduction

With rapid business iteration and evolving technologies, teams face numerous challenges in continuous integration and delivery, such as measuring performance, integrating cloud‑native and low‑code technologies, improving automation efficiency, aligning front‑end and product teams, and achieving truly continuous quality assessment.

Background

The information‑flow product serves personalized content to millions of users through a massive recommendation system comprising hundreds of modules, strategies, and models, updated at a rate of nearly one hundred releases per day. Achieving such speed and stability requires a tightly coordinated intelligent delivery system involving PM, RD, QA, and operations.

Core Issues & Solutions

The intelligent delivery system covers the entire lifecycle—from development and self‑testing, through testing, release, to deployment—by designing an efficient delivery model and addressing problems in each stage:

Development & Self‑Testing Stage – Introduce a micro‑service‑based business framework and execution engine to lower development cost, improve testability, and enable left‑shift testing.

Testing Stage – Decompose testing into input, execution, analysis, and定位, and apply data‑driven algorithms to boost efficiency and recall.

Release Stage – Evaluate release readiness automatically and flow qualified changes to deployment without manual intervention.

Deployment Stage – Optimize compilation, package trimming, intelligent monitoring, and dynamic concurrency adjustments through joint efforts of OP, EE, and RD.

Business Framework & Execution Engine

A two‑part architecture separates core execution from execution strategies. Core execution supports concurrent multi‑path short‑circuit execution, stateful/stateless operators, and flexible data handling. Execution strategies collect runtime metrics, periodically analyze them in independent threads, and generate execution modes using a plugin‑like system.

New R&D Mode Pilot

Developers perform autonomous testing by configuring cases or writing custom validation functions, reducing case‑writing time to under 30 minutes. QA provides a configurable automated testing framework covering >90 % of core functionalities, enabling low‑cost, high‑coverage testing.

Smart Test Input Generation

White‑box analysis of incremental code combined with business policies automatically generates test cases that maximize coverage of new features.

Intelligent Build

Using a smart build platform, the system evaluates task relevance based on white‑box analysis, historical results, and business characteristics, allowing selective execution of pipeline tasks and improving overall efficiency.

Performance White‑Box Analysis

Long‑tail degradation interception via a global performance analysis system (Dapper) integrated with offline performance tests and algorithmic decision‑making.

Variance elimination by correlating Dapper latency logs with function call chains to identify and correct abnormal spikes.

Unmanned Process Flow

From requirement entry to deployment, a risk‑assessment model evaluates change impact and guides automated stage transitions, enabling end‑to‑end unmanned delivery.

Results & Impact

Through the integrated framework, testing‑in‑the‑loop, intelligent pipelines, and automated flow, Baidu achieved over 50 % of demands delivered within a day, maintained stable or improving quality, and significantly increased throughput while reducing manual effort.