Automated Regression Testing for Advertising Recall: Tools, Process, and Lessons Learned

Manual Testing of Ad Recall

Ad business includes ad delivery, retrieval, and billing; this article introduces the role of testing throughout the recall process.

First, we outline the end‑to‑end flow from a page request to ad display and billing.

Manual testing has several drawbacks:

It depends on the stability of the APP service; if the APP or service crashes, testing becomes impossible.

The process is long, requiring constant page switching and log inspection.

Recall result accuracy is hard to guarantee, especially with complex strategies and ranking rules.

Recall and billing logs contain many parameters that must all be verified.

First Version of Ad Recall Testing Efficiency Tools

To reduce dependencies and streamline testing, the commercial testing team developed four tools: Recall Comparison, Recall Log, Recall Result, and Recall Case.

Recall Comparison Tool – compares two environments (e.g., dynamic vs. stable sandbox) to see the impact of strategy changes.

Recall Log Tool – presents a concise view of a single recall log, avoiding the need to read raw server logs.

Recall Result Tool – allows recall and billing checks without using the APP page; just select an environment and input ad parameters.

Recall Case Tool – maintains common test case templates, providing ready‑made ad parameters for each recall.

These tools shift testing from fully manual to semi‑automatic, removing reliance on upstream services and making results more intuitive.

Higher Requirements Beyond Individual Tools

Although each tool is handy, they do not cover the entire testing workflow; the process remains partially manual because recall results and logs are handled separately. After discussions with developers, a new solution was created that integrates both via interfaces, enabling more seamless regression testing.

Automated Regression Testing Solution for Ad Recall

1. Understanding the Core of Advertising

The advertising lifecycle consists of recall → strategy → scoring → ranking → billing. The core components for testing are recall, billing, and strategy. The team identified key test scenarios based on this core.

2. Core Cases Selection and Regression Process

Commercial ads are divided into seven recall‑source categories. For each source, a representative ad slot is defined, and validation points are extracted: Regular validation – checks whether the ad meets basic recall conditions (e.g., promotion time). Special validation – handles unique slots with additional checks. Billing validation – verifies click‑billing logic. Each case goes through four modules: parameter validation, regular validation, special validation, and click‑billing validation. After all modules pass, a final pass/fail result is produced, and failures include detailed step information.

3. Problems Encountered and Solutions

(1) Anti‑fraud strategy blocks billing validation – solved by replacing user information for each test case, ensuring unique credentials and limiting the number of billing checks.

(2) Data quality issues – solved by a data‑governance tool that batch‑processes problematic records before regression execution and performs pre‑checks on special cases.

(3) Special‑scenario cases cannot be validated – solved by a data‑construction framework that generates required test data on demand, making the cases reusable and extensible.

(4) Inter‑case interference causing failures – solved by decoupling cases, diversifying parameter selection, and avoiding shared categories that lead to cross‑impact.

Continuous optimization has raised the tool’s success rate to about 90%.

4. Effect Demonstration

The regression tool is used before each ad‑core service release to guarantee that core processes (recall, strategy, billing) remain unaffected. It also helps detect configuration drifts in pre‑production environments, such as mismatched ad‑slot settings or stale data, thereby improving testing efficiency and issue diagnosis.

Our Story Is Just Beginning

These challenges motivate ongoing enhancement of the tools, driving full‑automation of the advertising workflow, freeing human effort for higher‑value tasks, and continuously expanding test coverage.