Evolution of Baidu's System-Level Testing Platform

According to the classic testing pyramid, product testing is divided into three layers: unit tests, integration (module) tests, and system‑level tests. While lower layers provide broad coverage with low cost, system‑level testing, though expensive, catches critical bugs that have high impact and remediation costs.

Complex products such as search and advertising rely heavily on system‑level testing to ensure stability; a single performance defect can affect millions of users and cause significant revenue loss. As product architectures grew more complex, the demand for advanced system‑level testing and automation increased.

Ba​idu's system‑level testing technology has accumulated over six years of experience, undergoing several major architectural evolutions:

Manual Testing Era

Before 2009, testing was performed manually using tools like Apache's ab and LoadRunner, with results recorded in Word documents. The process was low‑efficiency but sufficient for the small, simple systems of that time.

Automated Script Era

Starting in 2009, Baidu introduced one‑click environment deployment, pressure testing, and automated report generation, dramatically improving efficiency. However, the proliferation of tools increased learning and configuration overhead, and resource contention grew with parallel projects.

Automated Platform Era

To address tool redundancy and management challenges, a three‑layer platform (test platform, environment, task) was built in 2011. The platform unified resource scheduling, provided visual configuration, and abstracted tool interfaces, allowing consistent control across diverse testing tools.

The platform introduced environment decoupling, splitting products into sub‑systems with drivers and stubs, and used traffic recording/replay to achieve near‑realistic test results, albeit with increased management complexity.

Testing Cloud Era

From 2011 onward, continuous integration increased the frequency of system‑level tests, demanding higher resource efficiency. Baidu integrated cloud technologies, consolidating resources, using HDFS/NFS for intermediate data caching, and employing incremental storage for fast snapshot and image recovery.

Key improvements included moving recording environments online, using the internal NB tool with Redis for data replay, and enabling arbitrary system composition via replay services.

Platformization Era

In 2014, under the company’s “interface‑oriented, platform‑oriented” strategy, the testing platform was rebuilt to expose testing capabilities as services. It now offers a generic environment provisioning component (supporting physical, virtual, and container resources) and a system‑testing component that can be configured for performance, diff, and load testing scenarios.

The platform provides a topology modeling tool where users define modules, sub‑systems, and topology templates, specifying resources, static parameters, and custom attributes. Users simply select a template, build the environment, and launch tests, while the platform handles scheduling and report generation.

Today, the platform supports dozens of products, executing nearly 10,000 system‑level test tasks per week.

Summary and Future Outlook

The three major reconstructions of Baidu’s system‑level testing—from product‑specific solutions to a company‑wide platform—are guided by three principles: staying up‑to‑date with product needs, maintaining a craftsman spirit for incremental improvement, and fostering an open‑source mindset to broaden the ecosystem. Future challenges include adapting to micro‑service and containerization trends, further complicating environment deployment and testing, and extending testing to online scenarios.

With a commitment to continuous innovation, Baidu aims to keep advancing its system‑level testing technology.