How Youzan’s Testing Team Supercharged API Automation for Faster, More Reliable Releases

Introduction

API automation testing is a common practice in most testing teams, but this article returns to the fundamental value of automation by sharing the Youzan testing team’s journey and insights.

Value

The team’s mission is to build an efficient and reliable product delivery capability, using various tools where API automation is a key weapon. To maximize its value, they evaluate automation quality based on two goals: increasing regression test efficiency and eliminating more testing blind spots.

Business Server Architecture

The system consists of multiple client channels (Web, H5, Mini‑program, App, Pad) reaching a Nginx gateway, which forwards requests to Web servers (Node.js) and backend Java micro‑services (API1, API2, Service1, etc.) communicating via Dubbo.

Two main options for API automation were considered:

Simulate client HTTP requests – fast coverage of user scenarios but requires extensive data creation and high maintenance.

Invoke services via Dubbo – allows mocking dependent data, but has higher script‑writing cost and cannot run in pre‑release environments.

The team combines both approaches to leverage their strengths.

Improving Regression Test Efficiency

Efficiency is addressed in three phases:

Before Regression

Precise coverage scope : Instead of relying solely on functional test cases, coverage is measured by exposed API count and backend service code coverage using JaCoCo‑based metrics (instruction, branch, cyclomatic complexity, line, method, class coverage).

Efficient script authoring : A custom Chrome plugin captures XHR requests, formats them, and allows batch upload to the testing platform, reducing manual data collection.

During Regression

With over 2,000 APIs and 10,000+ requests, full execution took more than an hour. To cut execution time to under 10 minutes, three techniques were applied:

Delay queue : Replace sleep‑based checks with delayed validation queues, allowing custom validation points.

Concurrent multi‑module execution : Use Java’s CompletableFuture with a thread pool tied to account resources, enabling parallel module runs.

Data cleanup via command pattern : Each cleanup task inherits from an abstract class; tasks are queued and executed after all test cases finish, ensuring reliable resource cleanup even on abnormal termination.

After Regression

A comprehensive, interactive test report is generated to streamline result analysis.

Eliminating More Test Blind Spots

The team analyzes missed bugs and identifies two main blind spots: intermittent data inconsistency and complex user‑scenario coverage. To address them, they extend automation to production environments.

Online Business Validation

Using the BCP business validation platform, they monitor backend topics, trigger API checks upon message consumption, and verify data consistency for each transaction, such as coupon delivery after a purchase.

Traffic Recording and Replay

They adopt Alibaba’s open‑source JVM AOP sandbox to record live traffic and replay it in test environments. The process includes:

Deploying an agent (Sandbox + custom plugin) as a Java Agent to capture traffic.

Recording entry calls and downstream calls (Dubbo, NSQ, MyBatis, Redis, HBase) with trace IDs.

Generating an MD5 hash for each trace to detect duplicates and create or update test cases.

Replaying recorded traffic against the tested code, supporting mock Dubbo consumers and middleware calls.

Comparing responses via JSON diff to detect bugs.

Compared with traditional API automation, traffic recording offers higher coverage of real user scenarios, automatic test case generation, and modular mock testing.

Conclusion

The evolving solutions presented aim to boost regression speed, broaden scenario coverage, and reduce blind spots, providing practical guidance for large‑scale API testing.