How to Systematically Test and Monitor AI Models in Large‑Scale Production

Background

Artificial intelligence is a key component of the ABC strategy at Autohome, and the rapid growth of AI services has created increasing challenges for the support team, especially in quality assurance. The supported projects now provide more than ten prediction models for lead ordering, conversion, and potential customer mining, compared with only a few CTR/CVR models at the beginning.

The data flow for these models involves three‑stage data sources from various product lines, collected offline and in real time, followed by preprocessing, feature generation, and finally model inference. This pipeline requires large‑scale real‑time and batch computation, diverse storage, and careful handling of feature accuracy, model performance, and service stability.

Testing before deep diving into the overall prediction business includes functional logic testing, performance testing, coverage testing, and data‑accuracy testing (the latter performed by manual sampling). Model effectiveness is guaranteed by developers, while testing is largely manual, leading to several challenges as testing depth expands.

Challenges

Model effectiveness testing lacks a systematic methodology.

Feature calculation logic is numerous and complex.

Data pipelines are long, massive, and time‑sensitive.

Online features and model performance lack monitoring for timely feedback.

Automation requires deep understanding of the entire computation chain and model strategies.

Test coverage points increase geometrically.

Solution

3.1 Model Effectiveness Testing

Method

Model effectiveness is evaluated through generalization ability using the following metrics: accuracy, precision, recall, F1‑score, AUC, ROC, and NDCG.

Accuracy (or precision) is suitable for binary and multi‑class tasks and represents the proportion of correctly classified samples.

In imbalanced scenarios, accuracy can be misleading. For example, with 90 % class A and 10 % class B, a classifier that predicts everything as class A achieves 90 % accuracy but provides no value for class B. Precision and recall are more appropriate for such cases.

Definitions:

TP (True Positive): predicted true, actually true.

TN (True Negative): predicted false, actually false.

FP (False Positive): predicted true, actually false.

FN (False Negative): predicted false, actually true.

Precision = TP / (TP + FP)

Recall = TP / (TP + FN)

Precision and recall are often inversely related; the F1‑score balances them, while the β‑parameter allows weighting recall (β > 1) or precision (β < 1).

ROC curves plot the true positive rate (TPR) against the false positive rate (FPR). If one ROC curve completely encloses another, the former model is superior. When curves intersect, the area under the curve (AUC) is used for comparison—higher AUC indicates better performance.

AUC reflects global ranking, whereas NDCG focuses on weighted ranking, making it more suitable when top‑k accuracy (e.g., top‑3) is critical.

3.1.2 Results

Large volumes of real samples are automatically evaluated against the metrics, producing detailed test reports that compare model outputs with developer baselines. The results are exposed via APIs, enabling simultaneous evaluation of multiple models, reusable evaluation scripts, and efficient regression testing.

3.2 Data Testing

Method

Data Sampling

Simple Random Sampling: randomly select m items from a population of n, giving each item equal probability.

Systematic Sampling: pick items at fixed time intervals (e.g., one record every 5 seconds) until the required sample size is reached, creating a random‑start systematic sample.

Reservoir Sampling: suitable for streaming data where N is unknown and data can be read only once; maintains a random sample of K elements from an ever‑growing stream.

Data Accuracy Test Method 1

Using the sampling methods above, a large set of samples is drawn to understand the end‑to‑end computation chain (see Figure 4). A feature‑calculation logic that does not involve storage format conversion is implemented (Figure 5). Expected results are generated and compared with the developers' outputs, producing a data‑test report.

Data Accuracy Test Method 2

After sampling, automated test cases compare the latest development‑stage service outputs with online services and related wrapper services (Figure 6). The results are integrated with the test‑cloud query‑diff tool to further improve automation efficiency.

3.2.2 Results

Automated data‑accuracy testing covers the entire computation pipeline—from source, storage, processing, to interface layers—without being affected by functional iterations or service encapsulation. Reusable scripts simplify regression testing, and large‑scale representative sampling uncovers bugs more comprehensively.

3.3 Automated Testing

Method

Data, model, and interface tests are all automated. Sampling algorithms actively draw massive real data, and validation modules for data, model, and functionality are built. Generic interface test scripts verify parameters (null, type, valid/invalid, existence) and generate default values, supported types, and nullability (Figure 7). Exception value lists are auto‑combined to create request permutations. Expected outcomes are defined, and results are automatically labeled as pass, need_verify, or fail.

Results

Reusable test scripts accelerate testing and regression cycles. Parameter‑combination scripts eliminate manual case writing, reducing a typical five‑day test cycle to half a day or one day. Iteration rounds increase from 1‑2 to 3‑5, and online quality improves noticeably.

3.4 Online Monitoring

Method

To ensure correct online data and model outputs and to detect issues promptly, monitoring of data accuracy, timeliness, and model‑effectiveness metrics is essential. Coverage, accuracy, model‑effectiveness, and reasonableness are monitored for offline tables, service logs, and API responses at minute, hour, day, and week granularities. Alerts are sent via email and SMS.

Monitoring scripts are integrated into the Autohome test‑cloud platform, providing unified management and visualizing trends and details (Figure 8).

Results

Since deployment, the monitoring system has repeatedly issued timely alerts, enabling rapid problem localization and resolution, thereby ensuring stable service for business users.

Future Plans

Future work will expand test coverage breadth, deepen test depth, and deliver more efficient testing solutions and simpler methods to further improve productivity.

References

[1] Zhou Zhihua. Machine Learning . Beijing: Tsinghua University Press.