Implementing Interface Diff for Automated API Testing with Jenkins and Pytest

1

Background

As business volume and complexity increase, each iteration requires extensive regression across impact domains, leading to low efficiency.

Business refactoring results in incomplete coverage and reduced quality.

There are many TCP interfaces, while HTTP interfaces are closer to business scenarios.

Given these characteristics, an interface‑diff method was chosen to improve testing efficiency and quality.

2

Implementation Goals

Compare interface data based on environment configuration and identify differences.

Integrate test cases using CSV files for unified management across business lines.

Integrate with Jenkins to automatically download code, execute cases, and generate reports.

Provide a unified entry point in the open platform, allowing selection of specific business lines, execution of corresponding case sets, and centralized viewing of reports.

3

Technical Implementation

Because test cases run on a server via Jenkins, CSV files are used for case management; each business line has its own CSV containing interface name, environment, request parameters, cookies, and related UID. Request parameters can be initialized from a data platform, while custom data may require packet capture.

The interface data set is initialized by selecting the appropriate business line in the platform, reading the corresponding CSV, and obtaining UID‑based PPU values from the data platform to prepare cookie data and assemble parameterized inputs for the test framework.

Pytest executes the parameterized cases, sends requests, receives JSON results, and forwards them to a service platform for diff comparison. Assertions compare differences, typically between online and sandbox results.

Unstable servers caused occasional diff failures; the pytest‑rerunfailures plugin was added to retry failed cases automatically.

After execution, a report file is generated and emailed; the platform page allows viewing detailed results.

4

Experience & Reflections

Interface diff mainly targets read‑type requests; it can be applied to write requests, but care must be taken to avoid generating dirty data in production and to verify reads after writes.

Cookie data must be configured per interface, as client logic varies with different v and t values.

Interface diff is similar to interface testing; comprehensive coverage of parameter combinations and edge cases is essential and complements functional testing.

Typical scenarios for interface diff include API refactoring, rapid feature iteration, and sandbox regression verification.

Since diff compares two environments, the source data must be consistent; data correctness is critical for reliable comparison.

Interface diff has limitations for newly added interfaces.

Current implementation compares all returned data and filters only changed fields (e.g., metrics). As case volume grows, diff time increases, so filtering strategies and concurrent execution need to be considered.

Unstable services often cause interface exceptions during execution, prompting the addition of a retry mechanism.

Long press to scan the QR code and follow the ZuanZuan QA team