Comprehensive Testing Strategies for Advertising Recall Systems

Advertising recall is a core component of commercial systems, frequently iterated to meet business needs, and even minor instability can directly affect revenue, making a thorough recall testing plan essential.

Demand Type Analysis

The ad recall requirements can be divided into three categories:

Adding ad placements – only business service changes, with little impact on the recall or search/recommendation services.

Recall and ranking strategy changes – modifications to the ad recall service (e.g., sorting rules, fill strategies, product types) that are invisible to the business side.

Underlying search/recommendation service changes – fundamental logic adjustments that affect recall results without the business noticing.

Testing Focus

Because the ad list parameters exceed twenty items and each influences billing and CTR, traditional interface testing is insufficient; instead, tests must verify that returned ads satisfy business query conditions.

For the three demand types, the testing emphasis differs:

Type 1 – validate business‑side request parameters (IP, UA, token, pageNo, pageSize, etc.) and ensure returned ads carry correct billing flags and placement order.

Type 2 – confirm that the recall logic changes produce the expected ad ordering and that logging (embedding points) is accurate.

Type 3 – check basic recall logic such as city and time constraints. Types 1 and 2 dominate commercial scenarios.

Solution

The proposed testing scheme addresses all three demand types.

Log comparison is performed by defining a standard log, then capturing request parameters and embedded logs during each test and comparing them to the standard, which validates both input completeness and logging correctness for the first two demand types.

The recall result tool allows testers to configure key parameters (slotid, city, category, keywords, etc.) from a stored set of real‑world parameters, invoke the ad recall service, and display the returned ads with clear indicators of whether basic information (city, time slot) meets the recall criteria.

Recall comparison quickly shows differences in results when the same parameters are processed by different code branches, providing an intuitive view for regression testing of various ad placement logics.

Finally, after validating individual cases, batch regression across other ad placements is performed using an RPC service diff tool, which offers broad coverage of scenarios.

Future Work

During practice, it was found that the recall result and comparison tools work well for single‑case tests but lack batch execution capability, limiting regression coverage. The roadmap includes building scenario‑specific case sets (offline, sandbox) for each ad placement, forming a complete suite that covers input logs, recall results, comparison, case regression, and batch diff, thereby fully controlling online ad recall quality.