From Fixed Inspections to Trusted AI-Driven UI Testing: AliExpress's Evolution

The article introduces the exploration and implementation of AI in AE UI testing, serving as a guide for subsequent series. It describes how the team transformed a tool primarily used for fixed inspections with manual step maintenance into a system supporting new requirement validation, multi-entry execution, and result governance.

Initial State: Execution by AI, Steps Maintained Manually

Fixed inspections accumulated capabilities but left a maintenance burden. aiStep describes page entry, element operations, and result judgment. AI participates in execution, but translating business intent into reliable steps still requires human effort. Maintenance pressure appears with changes: page entry adjustments, interaction redesigns, and copy changes require step re-checks. Each failure needs human judgment — real bug versus locator, data, or step issue. If every new requirement needs manual aiStep writing, scaling is hard.

Testing New Requirements Is Hard Because Multiple Preconditions Must Hold

"Testing new" means validating continuously changing new requirements, different from repeating stable inspection cases. It requires accurate requirement identification, coverage of main scenarios with clear preconditions and expectations, data supporting target states (account, region, permissions, environment), execution completing key actions, and assertions verifying business results. Any broken link makes the report lose value. Example: verifying an entry appears under a specific order state fails if the account lacks that order. "Check page normal" lacks specifics on required information and correct states. The execution model is only one link; requirement context, case quality, data construction, environment preparation, resource scheduling, and result judgment are all needed.

Build Test-New Pipeline First, Then Reuse for Inspections

The focus was on connecting the test-new main pipeline: receive requirement and case context, identify execution scenarios, fill necessary parameters, auto-generate or rewrite aiStep, and hand to execution. Generation must expand implicit premises into executable operations and observable expectations. Example: "verify receipt result" must answer where to receipt, pre-state, and evidence after click. Entry capabilities were made into Skills and integrated with the AIQ platform. Users can trigger from test plans, tasks, existing case instances, or natural language. The unified execution Skill organizes seven modes: plan execution, task execution, instance execution, one-sentence execution, UI Diff, and local step feedback. Existing inspection cases reuse rewriting, diagnosis, and controlled repair, shifting maintenance from manual per-case to auto-handling common changes with human review of exceptions. Generation success is only the entry point; business intent correctness is not guaranteed by generation success.

Separate Frequently Changing Parts

Architecture layers: Skill and platform entries organize user input, confirm parameters, submit tasks, and query status. AI test orchestration service handles step generation/rewriting, executability check, post-execution judgment, Mock calibration, and self-healing. PC execution uses AE-customized capabilities based on open-source Midscene; App side is handled by the independent SmartUI execution system. Execution resource scheduling service manages resource scheduling, capacity, and log analysis. Benefits: adding a business entry doesn't require rewriting the executor; adjusting failure classification doesn't change user task submission. Models, rewrite strategies, and business adaptations change but stay within boundaries. Important constraint: receiving a task ID only means the request was accepted; the task may not enter execution, finish, or have a retrievable report. The entry layer must not prematurely tell the user "success".

After Framework Runs, Use Metrics to Find Problems

Defined metrics: execution success rate, system success rate, usage rate, and result confidence. Each dimension addresses a different question and has specific misinterpretations to avoid:

Execution success rate : within clear executable scope, how many cases complete verification and pass. Misinterpretation: filtered proportion rise ≠ original requirement coverage increase.

System success rate : whether the service chain (submit, orchestrate, schedule, execute, return results) is reliable. Misinterpretation: correctly detecting product defects should not count as platform failure.

Usage rate : whether target users and applicable requirements truly use and continue using the tool. Misinterpretation: entry count and call count cannot replace effective usage.

Result confidence : whether pass/fail conclusions have sufficient evidence and withstand review. Misinterpretation: model self-evaluation high score ≠ result trustworthy.

For usage rate, the team cares whether business keeps the tool in actual workflows, not just tried once. For confidence, they ask: do passed cases really verify the target? Do failed cases really have corresponding issues? Clear sample selection, review standards, and uncertain-result handling are needed. These metrics help distinguish: system didn't finish, case didn't pass, system correctly found defect, system gave wrong conclusion — each needing different improvement actions.

Team Parallel Exploration to Fill Gaps at Different Levels

Based on user feedback, the team explores in parallel:

Self-updating knowledge base : precipitate page info, operation experience, and jump relations observed during execution, letting subsequent tasks leverage existing understanding and reduce repeated probing. Current engineering has page knowledge reporting, aggregation, and query, but integration across execution entries varies; knowledge needs expiration and conflict handling — "recorded" cannot be treated as "always correct".

Agent autonomous planning : for a natural language task, break into todo steps, assign deterministic prep actions (login, region/language) to functions, then organize UI operations; replan on execution obstacles. Increases execution flexibility but adds a new check: whether planning deviates from the original goal.

Rewriting and self-healing adapted to AE business : first make case descriptions more execution-friendly, then combine failure evidence to judge repairability, verify after repair, and reuse experience. Environment issues suitable for retry, step issues needing modification, and product defects requiring human confirmation cannot share the same rerun strategy.

Directions complement but must validate on concrete samples. More knowledge, flexible planning, and aggressive repair don't automatically equal more reliable results.

Four Strategy Adjustments Shaping Today's Tool

Adjustment 1: Identify Non-Executable Reasons First

Initially used final execution success rate to evaluate. Poor results led to "execution capability insufficient" conclusions. Breaking down failures revealed many occur before execution: data not constructed, account lacks target state, case description missing key conditions, or content unsuitable for UI-only observation. Rerunning these only consumes device and model resources. Strategy contraction: first identify missing preconditions, mark cases as non-executable with reasons, prioritize cases with conditions. After conditions filled or description corrected, re-evaluate executability. Warning: avoid using filtering to inflate report numbers. Current report calculates executable pass rate as passed / (passed + failed), where passed = judged pass, failed only includes product defects and environment issues; some other failures are reclassified as non-executable after judgment. Showing a single ratio hides what was excluded. Need to retain original total, pass, fail, and non-executable counts, and separate "pre-execution exclusion", "post-execution rejudged non-executable", and "never ran". The former may need data construction, the middle may need case rewrite or Mock, the latter needs execution chain investigation. Code includes this split, letting the report explain next steps. Executable success rate and executable coverage must be viewed together : the former improves runnable parts, the latter reminds how many requirements are not yet caught by the tool.

Adjustment 2: Constrain Divergence, Ensure Pass/Fail Have Evidence

Focusing on executable cases, the team encountered AI giving seemingly reasonable execution but not achieving the real test goal. Example: goal is to verify page state updates after claiming a benefit. If execution only confirms "clicked claim button" and "toast appeared", it may miss claim failure or state not refreshed; conversely, slow async refresh may be misjudged as a product issue. Hence the focus on "false pass" and "false fail". Judgment cannot rely solely on execution endpoint state or model re-reading its own explanation. Must align original expectation, key actions, assertions, same-execution logs and screenshots; in Mock scenarios, verify expected rules actually hit. When evidence is insufficient, allow result to be temporarily uncertain rather than forced pass. Repair must follow same constraints: corrections to locators, waits, step expressions, Mock configs need justification; deleting key assertions or bypassing necessary operations cannot be used to boost success rate. This is the design discipline for constrained self-healing, needing continuous validation. Engineering has step snapshots, post-repair verification, and failure rollback protections, but repair may modify assertions or rewrite steps entirely, so must check rewrite preserves original business intent. Human judgment still needed for real defects and ambiguous business expectations.

Adjustment 3: Frontend Migration Requires Different Test Condition Preparation

Later, collaborating with a business domain on test-new, data issues again became the main blocker. After filtering, remaining executable cases were limited; waiting for real data preparation limited tool value. Key condition: business was mainly doing frontend migration. Many cases verify that under given data states, the new page's display and interaction meet expectations. For such goals, Mock can control response data, letting the page stably enter normal, empty, error, or specific business branches. Added Mock verification system: not just intercept requests, but also check interface match, response structure compatibility, scenario values match case, and rules actually hit. Engineering probe calibration uses captured real requests and responses to assist correction, while preserving test scenario needed values, avoiding changing originally verified abnormal states to normal.

Migration also requires new/old feature consistency, so added UI Diff entry and diff verification capability, letting new/old page comparison enter the workflow. Helps locate display and interaction diff clues; different execution chains' comparison capabilities have boundaries, cannot equate one entry with full coverage of all pixel, behavior, interface diffs. Two capabilities division: Mock prepares controllable scenarios, diff comparison discovers migration changes, business assertions check explicit test expectations. Original page may have defects, so "consistent with old page" still needs judgment in requirement context. Mock pass ≠ real backend, payment chain, production data correct; no page diff found ≠ all functions fully equivalent. End-to-end verification scenarios still need real data and real chains.

Adjustment 4: Let Entries Adapt to Users, Not Vice Versa

Collaborating with more business domains revealed different starting points for UI automation: some have complete test plans wanting batch execution; some have only a business action wanting quick verification; some are familiar with page operations but unwilling to write steps from scratch. Added browser recording and one-sentence execution capabilities. Online recording is the user actively operating on the business page and precipitating steps, not passive collection of online user traffic. One-sentence execution still needs necessary account, device, and environment parameters; it reduces task organization cost, doesn't make preconditions disappear. Understanding of "unified entry" changed: can have multiple input methods, but parameter checks, execution status, and result evidence should be as unified as possible. Users need not conform to a single starting point; the platform remains responsible for delivered results.

After Rapid Implementation, Must Quickly Prove Changes Effective

The author submitted 200k lines of code cumulatively around architecture upgrades, capability iterations, and new features. This is cumulative commit volume for transformation work, not execution engine code size or current repo net lines. The whole system is built on open-source projects, existing engineering, and team collaboration. AI speeds up implementing ideas, but also makes verification pressure more prominent. Modify step generation → check coverage change; adjust failure classification → confirm real defects not misclassified; add self-healing strategy → check if it turns wrong results into "pass". Manual full runs and log reading each time can't keep up with change speed. External models and tools also continuously change; a tool may gradually lag behind new capabilities and business requirements even without code changes. This means building "quickly propose and implement a change" and "quickly judge if change effective" together.

Shorten Feedback Loop First, Then Gradually Improve Evaluation

Built offline data pipeline analysis for the UI execution tool, linking tasks, steps, execution logs, judgments, and repairs to locate the problem segment. "Offline" means analysis and review detached from a complete online UI execution process, not fully offline nor no model calls. Some issues can be validated on historical logs and small samples first: e.g., modify classification rules → check if existing failures correctly distinguished; replace rewrite strategy → compare if steps drop key expectations, then small-scale real execution verify. Only parts involving real page state, device timing, or interface interaction must confirm via corresponding execution chains. Architectural generalization serves this: entry separated from executor, judgment and repair decoupled, business context passed via explicit inputs, allowing strategy replacement, multi-scenario access, controlled experiments. Otherwise each attempt requires changing the entire chain, and verification cost explodes. Current evaluation components: case generation side has golden set comparison scoring; execution engineering has localization, planning evaluation entries, and log classification regression tests. But these components cover different objects, cannot be pieced into a mature end-to-end UI evaluation platform. Especially result confidence still needs stable samples, independent review, and continuous regression. Next, the hope is to fix an iteration path: select samples from real failures, clarify hypothesis to verify, quickly compare in offline-checkable parts, then confirm gains and side effects via small-scale execution. Strategies not passing verification are not expanded just because they are easy to implement.

Stage Results Backed by Resources, Scheduling, and Business Adaptation Working Together

Two Metrics >90%, Each Answering a Question

In current Mock test scenarios, observed execution success rate and result confidence both exceed 90%. These are independent dimensions, not a composite score. Execution success rate describes verification pass under controlled data conditions within executable caliber; result confidence focuses on conclusion credibility, reminding to check evidence behind "green". Service chain stability is measured separately by system success rate. Caveat: these numbers come from the author's feedback on current Mock scenarios, not yet providing auditable statistical window, sample size, or independent review details. They don't represent all businesses or all original submitted cases; not for cross-platform ranking or full-business SLA; not used to derive requirement coverage or defect recall. Future evaluation topics need to clarify these calibers. For broader test-new scenarios, data dependencies and business judgments remain. Mock makes some frontend verification conditions controllable, but doesn't automatically solve other scenarios.

Real Devices and Servers Enable Exploration to Land on Real Execution

Hundreds of real devices and local servers allow validation on real device environments and provide a resource pool for multi-task execution. Resource utilization depends on scheduling: devices/servers have different capabilities, tasks have different browser, CPU, memory, and device occupancy requirements; high concurrency needs capacity judgment, allocation, queuing, and post-execution resource recycling. Execution resource scheduling service's node capacity management, task allocation, and log analysis handle these. The author values a sustainable optimization foundation over converting device count directly to concurrency. Hundreds of devices ≠ hundreds of tasks always runnable; actual throughput affected by device type, availability, task duration, and environment constraints. Without same-condition measurement, should not give a unified performance improvement multiple.

Business Domain Adaptation Lets Public Capabilities Enter Specific Processes

Different business domains have their own requirements for entry, account, region/language, page state, data, and assertions. General executor's page actions need to combine with these business premises to become a meaningful test. The team reuses public generation, execution, evidence, and scheduling capabilities; puts differences in the business adaptation layer. New scenario integration can gradually supplement missing capabilities instead of replicating the entire platform. Also means ongoing costs: device maintenance, knowledge updates, Mock contract calibration, regression after business rule changes. Ability to customize and iterate is an advantage; knowing long-term maintenance needs is equally important.

This Is a Guide and an Unfinished Answer

Reviewing the journey: initial problem was manual aiStep writing/maintenance being too heavy; after framework ran, used metrics to break down problems; after entering business, continuously adjusted the tool's scope and manner through executability identification, evidence governance, Mock, and multi-entry adaptation. For those preparing similar exploration, the most valuable experience: first look at failure samples clearly. Data prep insufficient → supplement data or adjust verification; results untrustworthy → supplement evidence and review; user onboarding difficult → improve entry. Iterate along actual blockers, easier to achieve verifiable progress than attributing all to "model not strong enough". AI test-new cannot fully replace humans yet. Requirement boundary reasonableness, key risk omissions, whether a difference is allowed, fund and compliance scenario validation still need business and test involvement. Automation reduces repetitive labor, but final quality responsibility cannot be handed to an unreviewed status value. Subsequent series will follow the actual chain: natural language to PC/App execution organization, self-updating knowledge base and autonomous planning, rewrite and self-healing boundaries, Mock and UI Diff migration verification practices, real device scheduling and offline evaluation. Each topic will bring concrete implementations, pitfalls, and unsolved problems. From fixed inspections to today, more scenarios can be caught, and requirements for "pass" are more concrete. Next steps: bring more real requirements into verifiable scope, and make every automated conclusion easier to check, explain, and trust.