How a Six‑Layer Capability System Enables AI Agents to Automate New UI Tests

Background

The AE business testing team integrated UI testing into development, making every new requirement a self‑tested feature. New UI tests must run in a pre‑release environment that changes constantly, with fresh test cases, accounts, configurations, and data, leaving no stable selectors or recorded scripts. This makes automation extremely challenging.

Why "testing new" is hard

Over two months the team delivered an end‑to‑end AI pipeline that generates test cases, executes them, and produces a trustworthy report. The biggest lesson was that AI‑generated test cases are not the bottleneck; the real obstacle is the 75% of configuration, environment, account, and data preparation that must be ready before execution, and the detection of "AI false‑passes" after execution.

Six‑Layer Capability System

The system is organized into six layers (C1–C6), each addressing a specific problem:

C1 – Case Generation

AI can generate textual steps quickly, but for "testing new" the generated steps must be directly consumable by downstream agents. The team introduced a rewrite engine that normalizes free‑form AI output into structured - action / - assert steps using a few‑shot prompt covering click, input, scroll, and assert actions. Multi‑source degradation (Spec → Diff → pageUrl → knowledge base → fallback) ensures an AIStep is always produced.

C2 – Pre‑Construction

This layer handles the 75% of cases that block execution: configuration changes (GCP, GOP, Switch, Diamond), multi‑dimensional environment orchestration (US‑East, US‑West, multiple countries/languages, dark mode), account provisioning, and data construction. Functions such as set_diamond(module_id, config), login, change_locale, and change_env are exposed via a Provider service, turning UI clicks into fast API calls (5‑8 UI steps reduced to 1 function call, cutting time from ~70 s to ~4 s).

C3 – Environment Orchestration

After C2 decides the target environment, C3 actually switches to it using the Provider service. Supported actions include deep‑link navigation ( ae://goto?url=xxx), account login, locale change, environment change, RTL layout toggling, dark‑mode toggling, text input, screenshot/recording, popup handling, and mtop request capture.

C4 – Self‑Planning Execution

On the app side (SmartUI) the agent receives the current screenshot and case information, decides the next action (click, input, scroll, etc.) via a three‑stage loop (Observation → Thought → Action → Re‑planning). The loop runs eight atomic operations ( aiTap, aiRightClick, aiDoubleClick, aiHover, aiInput, aiKeyboardPress, aiScroll, aiPinch) wrapped in a higher‑level aiAction cycle.

C5 – Assertion & Attribution

After execution, an assertion agent validates each step. The team discovered AI reports a 56% pass rate, but manual review showed only 19.5% were truly correct—a three‑fold inflation. To combat this, they defined nine strict rules (e.g., "pre‑condition not constructed", "single‑frame final state", "missing element = fail", "environment error", "backend payload mismatch", "agent stopped early", "unsupported platform", "assertion intent mismatch"). When any rule matches, the case is marked BLOCKED and execution stops, separating genuine AI failures from environment issues.

C6 – Reporting & Knowledge Feedback

Each run produces a comprehensive report: screenshots, video, mtop traces, logs, and per‑step evidence. The report also feeds back into three knowledge stores—business concepts for rewrite, page knowledge for planning, and page visit records for assertion—closing the loop so future cases benefit from past executions.

Measuring Capability

Instead of overall pass rate, the team uses a capability probe baseline. A fixed set of 100+ Android cases is divided into five buckets (basic UI, configuration & data, multi‑environment, account & experiment, data & API). Each bucket targets a specific layer; failure indicates the missing capability. Stability is enforced by requiring < 3% variance over three consecutive runs. Improvements are evaluated by delta tables per bucket, and a v2 strict‑review compares AI pass rate with effective rate to expose cheating space.

Practical Results

In a real AE homepage redesign, the six‑layer pipeline executed end‑to‑end, achieving an AI execution pass rate (including unsupported pre‑conditions) of XX% and an effective case execution rate of 85.7%. Two real defects were uncovered:

Price‑strike vs. original price mismatch caught by C5 rule 8 (assertion intent mismatch).

RTL arrow direction error caught by C2+C3 environment setup and C4 execution, with C6 providing side‑by‑side screenshots.

Key observations: defect density correlates strongly with multi‑dimensional combinations; style‑related bugs are the most common false‑passes; many cases still fail due to incomplete pre‑construction capabilities.

Lessons & Principles

AI can only contribute ~20% of total automation value; the remaining 80% comes from engineering the surrounding layers.

Reserve agent autonomy for tasks requiring semantic understanding (state recognition, next‑step decision, evidence judgment). All routine actions (environment switch, login, config injection) should be function‑driven.

AI false‑passes are systematic; maintain a dedicated "AI fake‑pass" rule set and a fast‑path BLOCKED handling to surface them.

These principles are applicable to any UI/Browser agent team seeking to scale automated testing for rapidly evolving features.