From a Single Command to a Full Delivery Loop: Designing and Practicing AliExpress Test Skill System

Background

After AICoding expanded to multiple business domains, development speed increased while testing became a bottleneck. Manual effort, high maintenance cost of automation, and limited coverage caused testing to lag behind iterations. Directly prompting large models with PRD text yielded only 60%‑70% adoption because generated cases were vague, contained hallucinated steps, and missed boundary scenarios. Moreover, analysis, generation, and execution stages operated in isolation without an end‑to‑end data flow.

Test Skill definition

Test Skill : a set of AI‑driven capabilities running on AE AAIC. Each Skill encapsulates a concrete testing function (e.g., case generation, unit testing, API testing, data construction) and is orchestrated by a unified engine into a full automation pipeline.

Architecture & Orchestration

The system follows a “one main pipeline, two execution environments, multiple extension slots” model. The unified entry point is /aaic:test. Nine atomic Skills are composed sequentially into an end‑to‑end flow: code‑change analysis → case generation → case storage → unit test → API test → UI automation → loss‑risk analysis → data collection → test report. Only the requirement ID and code branch are required; the rest is fully automated.

Local verification vs. Cloud regression

Local fast verification identifies the impact range of a change using code graphs and domain definitions (L1‑L4), then generates layered structured cases. Unit tests are compiled and executed locally; failing cases enter a reflective loop (execute → analyze error → infer fix → re‑execute) for up to five rounds, with coverage measured by JaCoCo.

Cloud regression integrates with the AAIC‑integrated test environment, triggers UI automation with pre‑constructed data, and reuses existing assets for full‑stack regression, covering scenarios impossible to validate locally (cross‑service calls, front‑end rendering, full‑link data flow). Both stages share a unified case identifier and merge results into a single report.

Domain‑specific extensibility

Design adopts “strong platform guarantees + domain customisation”. Platform‑guaranteed nodes (case generation, case storage, data collection) are immutable to preserve data integrity. Domain‑customisable slots are provided before unit tests, after API tests, and at the API test node itself, allowing domains to insert new steps or replace default Skills. Validated domain Skills can be promoted to platform defaults following the path “domain incubation → cross‑domain pilot → platform consolidation”.

Core capabilities

Test case generation

Feature extraction : use code graphs and L1‑L4 definitions to identify affected functionalities.

Test‑point derivation : match features against a business‑knowledge rule base and testing methodology to produce test points covering normal, abnormal, and boundary flows.

Case generation : cross test points with eight dimensions (interaction, backend logic, security, loss, performance, i18n, compatibility, instrumentation) to output fully executable cases.

Three artifacts are produced: a feature list, a test‑point list, and the final test cases.

Quality‑assurance layers

Layer 1 – Context compression : extract only test‑relevant information (interfaces, rules, state transitions) and filter out irrelevant markup.

Layer 2 – Business knowledge distillation : inject domain‑specific knowledge via a constructed knowledge graph, dynamic prompt injection, and rule‑based constraints to avoid hallucinated scenarios.

Layer 3 – Case recall : a two‑stage recall architecture (coarse text‑similarity screening followed by LLM‑level re‑ranking) ensures regression coverage for impacted legacy functionalities.

Effectiveness metrics

Adoption rate : test‑point‑driven generation achieves 90%+ versus 60%‑70% for conventional AI case generation.

Coverage : >90% versus <80% for the conventional approach.

Executability : cases are automatically labelled for downstream automation, eliminating the need for extensive manual completion.

Automated execution

Local execution : unit tests generated with business semantics are compiled and run locally; failing cases undergo a reflective loop of up to five iterations, with coverage reported by JaCoCo. API tests combine code semantics with recorded traffic to construct inputs.

Cloud regression : UI automation is triggered in the integrated environment, covering front‑end rendering and full‑link interactions that cannot be validated locally.

Data construction

The Data Construction Skill enables natural‑language‑driven creation of end‑to‑end test data across multiple domains (product, order, fulfillment, marketing, account). Its architecture has three layers:

Semantic understanding layer : LLM parses intent and determines required tools.

Tool orchestration layer : selects and sequences domain tools (e.g., publish product → add to cart → place order → pay → ship → dispute).

Execution layer : invokes tools via the MCP protocol; new domains only need to expose an MCP Tool.

Supported domains include product/merchant, order/payment, fulfillment/reverse, marketing/interaction, and account/address.

Data collection, analysis, and reporting

Each run records data at three levels:

Local archive : test cases, unit‑test summaries, execution status, and reports are stored locally for traceability.

Platform sync : cases and results are written to the AIQ platform under a unified reporting protocol.

Metric analysis : automatically aggregates case coverage, unit‑test coverage, execution pass rate, and productivity gain (person‑days saved). Low‑coverage modules are prioritized, and failing cases trigger automatic investigation.

Test report

Key metric overview (design, unit, API, loss‑risk).

Critical actions (failed cases, loss‑risk scenarios, pending executions).

Case tree with hierarchy, priority, and coverage statistics.

Detailed sections for each stage.

Evolution directions

Case‑generation quality and self‑evolution : incorporate multi‑source inputs, broaden knowledge distillation, improve recall precision, and close the loop by adjusting prompts based on execution outcomes.

Left‑shift and full‑domain rollout : adopt a pre‑code case‑first approach, provide post‑code incremental guarantees, and achieve pervasive adoption across developer workflows.

Addressing execution bottlenecks : co‑create data‑construction tools with domains, stabilise test environments, and expand the domain‑Skill ecosystem.