QA Quality Delivery Practice: Cutting Multi‑Scenario API Regression from 3 Days to Half a Day

Why scripts alone aren’t team capability

The original API automation suite was built before the Agent platform and was used for daily testing and large regression scopes covering many roles, interaction types, business scenarios, and time‑window conditions. Running the full matrix manually required about three days; after automating high‑frequency operations the same coverage could be completed in roughly half a day. Early scripts lived on individual developers’ machines with bespoke environments, parameters, and pre‑conditions known only to the author, preventing easy reuse across the team.

Four‑layer runtime structure and delivery chain

Unified entry layer : receives natural‑language or structured commands and provides a consistent external entry point, outputting a standard request.

Scene & parsing layer : receives the user expression and scene definition, identifies the scene, extracts parameters, checks for missing items and fills derivable parameters, outputting an executable plan or an explicit error.

Execution routing layer : receives the scene identifier and prepared parameters, maps the scene to an existing execution capability, and outputs a standard execution result.

Underlying interface capability : receives the determined interface parameters, performs real test‑environment queries or business actions, and returns the interface response with a result summary.

An independent delivery chain builds a Skill package that the Agent platform can install and run.

Scenario list to bound natural‑language execution

Only registered scenarios are accepted. A scenario must be high‑frequency, have standardizable parameters, and have a stable underlying interface. Currently 26 scenarios are registered, covering login, room queries, interaction operations, gift handling, and status checks.

Scenario contract includes:
- Scenario ID
- Supported expressions
- Required parameters
- Optional parameters
- Default/derivable parameters
- Corresponding execution capability
- Success/failure response format

When a user inputs a sentence, the system performs four steps:

Determine whether the input matches a registered scenario.

Extract parameters and normalize them to standard fields.

Check required fields; derive context parameters if needed.

Generate an execution plan; if no missing or conflicting items, proceed to execution.

If any step fails, a structured error is returned and execution stops.

Five CLI entry points for full visibility

scenes

: shows which scenes are currently supported (no real operation). help‑scene: displays the parameters required for a specific scene (no real operation). parse: reports which scene matched and which parameters were extracted (no real operation). plan: after filling derivable parameters, shows how the execution will be performed (no real operation). run: executes the prepared scene with parameters (triggers the underlying script).

Only run triggers the real script; the other commands are for discovery, validation, and planning.

Delivery pipeline selection

Constraints:

Development environments include both Windows and Linux; the Agent target runtime is Linux.

The Agent platform integrates Skills via a CLI interface.

No additional service deployment should be required.

The platform should consume a built artifact without needing direct access to the Python source repository.

Local ongoing changes must not affect the version used by others.

Four candidate solutions were evaluated:

Directly upload Python source code : most direct integration but couples source, dependencies, and runtime; lacks a release boundary.

Linux CLI binary : fits the existing CLI integration, requires no separate service, and allows the platform to consume only the built artifact.

Binary with fully external configuration : provides stronger isolation but was not the primary concern at this stage.

Separate service deployment : offers unified calling but adds deployment, runtime, and maintenance costs.

The team chose the Linux CLI binary approach and defined the delivery chain as:

Maintain Python source on Windows and Linux
↓ Unified CLI contract
↓ GitLab Linux Runner installs dependencies and runs automated tests
↓ PyInstaller builds Linux executable
↓ Assemble Skill bundle
↓ Download CI artifact and upload to Agent platform

CI first validates scene parsing, routing, protocol, and the five CLI entry points; only after passing tests is the Linux artifact generated and packaged for the Agent.

Execution flow and traceability

After integration, users invoke registered scenarios via business expressions; the Agent interprets the request, selects the appropriate Skill, and orchestrates the call. The scene & parsing layer restricts execution to registered scenarios, validates parameters, and surfaces errors. The underlying script performs deterministic test‑environment actions. Execution results are returned in a structured success/failure format, and business‑side records are left for later verification. The final test verdict remains the responsibility of the QA team.

Achievements

Consolidated scattered API scripts into a unified scenario entry.

Registered 26 high‑frequency scenarios with defined parameters and execution rules.

Standardized discovery, validation, planning, and execution via scenes, help‑scene, parse, plan, and run.

Established a CI‑driven, Linux‑build, Skill‑bundle delivery pipeline.

Validated multiple real‑world test scenarios within the Agent platform.

Next topic

The upcoming article will explore how to connect data, environment, and results after API execution.