From Feasibility to Self‑Evolution: Four‑Stage Intelligent UI Test Case Generation

1. Pain‑point Diagnosis

Rapid product iteration creates a mismatch: faster releases increase test‑coverage pressure while complex business rules widen the quality gap between senior QA and newcomers. Statistics show test‑case authoring consumes only 13% of effort but execution uses 38% of resources; knowledge is siloed, PRD changes are frequent, and defect knowledge is hard to reuse.

Key insight: The real bottleneck is not automation of execution but the source, quality, and reusability of test cases.

"Quality Gap" – Knowledge Islands

Senior QA writes comprehensive scenarios (e.g., concurrent duplicate‑payment protection).

Junior QA copies literal requirements and misses hidden rules.

This reflects a lack of systematic knowledge‑capture and reuse; each defect remains a personal asset.

2. V1.0 – Prompt Engineering (Feasibility)

Goal: prove that a foundation model can generate usable UI test cases from PRD/technical specs.

Few‑shot learning: select 3‑5 high‑quality examples.

Scenario‑specific prompts: different templates for functional PRD, technical specs, full‑stack docs.

Strong output constraints: enforce indentation, punctuation, line breaks for parsable results.

Workflow:

1) Input: PRD + few‑shot examples + scenario prompt
2) LLM inference (single agent, no external tools)
3) Output: structured test‑case set

Result: generation rate only 8%; feedback – “too generic, cannot use directly.”

Fundamental issues uncovered

Long‑document decay: when PRD exceeds ~30K tokens, the transformer’s attention degrades, causing earlier context to be forgotten.

Missing business knowledge: implicit rules (e.g., concurrent payment checks) are not inferred.

Black‑box generation: users cannot intervene in test‑point splitting or case structure, making correction cost higher than writing from scratch.

3. V2.0 – Multi‑Agent Collaboration

Observation: no QA writes a perfect case in one step; the workflow includes design, review, and iteration.

V2.0 orchestrates three specialized agents mirroring the QA workflow, adding a review loop at each stage.

AI responsibility: fast generation, provide structured scaffolding, handle parameter variations.

Human responsibility: critical review decisions, error correction, priority labeling.

Metrics improvement over V1.0:

Generation rate ↑ from 8% to 15% (87% increase).

Adoption rate ↑ from 40% to 55% (37% increase).

User penetration ↑ from 20% to 30% (50% increase).

New limitation: generated cases are correct but shallow.

"A single workflow split into multiple dedicated agents improves controllability and auditability, reducing end‑to‑end errors."

4. V3.0 – Knowledge Engineering (Business‑aware AI)

Example contrast:

Junior QA: input > balance → expect recharge prompt (✓).

Senior QA adds concurrent‑payment test, idempotency checks, network‑retry logic, etc.

Coverage gap can be tenfold; prompts alone cannot inject hidden business rules.

Solution: systematic injection of four knowledge assets:

Textual test material: PRD, technical specs, code diffs (front‑end information).

Multimodal material: UI prototypes (Figma), flowcharts, architecture diagrams – provide visual context.

Historical case knowledge (RAG): retrieve relevant defect records to guide generation.

Private business knowledge: rule graphs and team‑specific templates that turn experience into reusable rules.

RAG workflow:

Construct query (e.g., “live‑gift scene + historical defects”).

Retrieve top‑K relevant defect records.

Augment prompt with retrieved knowledge.

LLM generates richer cases.

Metric shifts (V2.0 → V3.0):

Generation rate ↑ from 15% to 35% (133% increase).

Adoption rate ↑ from 55% to 65% (18% increase).

User penetration ↑ from 30% to 50% (67% increase).

Historical defect coverage ↑ from 12% to 76% (533% increase).

Drawbacks: review cost remains high, knowledge‑maintenance effort grows (170+ private templates), and knowledge updates lag behind new business rules.

5. V4.0 – Agentic Self‑Evolution (Closed‑loop Quality Flywheel)

Core innovation: the system can autonomously discover, correct, and evolve.

Three pillars:

Autonomous decision & ReAct loop: after a user submits a task, the agent iterates “reason → observe → act”.

Autonomous review & critique: structured feedback dimensions (coverage, structure, hierarchy, naming) replace binary yes/no checks.

BadCase rule engine & self‑evolution: low‑quality cases are labeled, patterns extracted into rules, and automatically applied to future generations.

Asset hierarchy used in final output:

Skills: 170+ customizable skill packages.

Memory: user preferences, session history, context links.

Knowledge: business rule graph, historical cases, defect records.

Example flow:

User: “Mark priority, PO first”.

Memory records: userPreference = {PO cases top, priority required} Subsequent generation automatically annotates priority and reorders PO cases.

Review becomes multi‑dimensional (module‑level and case‑level) with criteria such as PRD coverage, scenario completeness, redundancy, step clarity, expected result definition, and priority rationality.

BadCase pipeline:

QA flags a low‑quality case.

System extracts failure pattern and stores as a rule.

Future generations auto‑invoke the rule for similar scenarios.

Rules aggregate into generic standards, continuously enriching the knowledge base.

"Scene‑rule template update time dropped from days to 5 minutes, reducing maintenance cost by 99%."

6. Intelligent Execution – From Generation to Reliable Run

Two traditional routes:

UI automation scripts + AI self‑healing – stable but brittle to UI changes.

Pure AI exploration – flexible but lacks clear test intent.

Kuaishou’s third path: “test‑intent‑driven” execution.

"Let the test case carry an explicit intent, and let the execution engine plan and run autonomously under that constraint."

Prerequisite 1 – AI‑friendly cases must be concise, atomic, observable, and explicit about environment dependencies.

Prerequisite 2 – Execution engine possesses a “four‑sense” loop (perception‑decision‑action‑feedback) that respects the intent constraint.

Resulting workflow (KATE platform): upload PRD → generate AI‑friendly cases → execute with intent‑driven engine → view consolidated report.

7. Four‑Layer Architecture Methodology

The evolution follows a continuous chain: information flow → feedback flow → knowledge flow → capability flow.

Layer 1 – Scenario Stratification

Not every feature merits AI generation; prioritize high‑impact, low‑complexity scenarios first.

Layer 2 – User Operations (BadCase Loop)

Low‑quality cases become improvement signals; each cycle accumulates experience and reduces repeat mistakes.

Layer 3 – Knowledge Operations (Automatic Sedimentation)

Automate the lifecycle of knowledge assets: capture, index, retrieve, and apply.

Layer 4 – Agentic Architecture (Sustained Evolution)

Dynamic system‑prompt assembly.

Rapid skill extension.

Memory‑driven user preference learning.

Knowledge auto‑sedimentation and retrieval.

8. Summary – Future Shape of End‑to‑End Intelligent Testing

Key takeaways:

Automation should replicate staged review rather than attempt a one‑shot solution.

Quality ceiling is set by knowledge richness, not prompt cleverness.

Systematic BadCase collection fuels self‑evolution.

AI will replace repetitive tasks but cannot supplant critical human decisions.

Industry parallels (NVIDIA HEPH, Google Firebase App Testing Agent, open‑source Test‑Agent community) confirm that the path from single‑agent prompting to multi‑agent collaboration, knowledge engineering, and agentic self‑evolution is a shared direction.

9. Practical Recommendations for Teams

Start with high‑value, low‑complexity scenarios to quickly achieve >50% generation rate.

Prioritize building a robust historical defect library before fine‑tuning prompts.

Design feedback loops (user feedback → problem classification → validation) from the first version.

Clearly delineate AI responsibilities (generation) and human responsibilities (review, knowledge upkeep).

Overall, Kuaishou’s four‑stage evolution demonstrates how AI‑augmented testing can transition from a fragile proof‑of‑concept to a self‑evolving, enterprise‑scale quality system.