How Test Engineers Can Thrive in the AI Era: Upgrading Skills and Roles

At the 9th AI+R&D Digital Summit, Kuaishou’s R&D Agent and Intelligent Quality Platform lead presented the evolution of AI‑generated UI test cases, showing a generation rate that rose from 8% to 15%, then 35%, and finally 70% over two years.

The low 8% rate was not due to model weakness but because the AI was fed only the PRD text. The later improvements came from three key changes:

V1 (8%): AI saw only the literal PRD, missing business‑specific risk context.

V2 (15%): Introduced human review to filter AI output, but still lacked systematic knowledge.

V3 (35%): Systematized senior QA experience into a knowledge base, allowing AI to retrieve historical defect patterns and business rules.

V4 (70%): Implemented a closed‑loop "BadCase → Rule → Auto‑correction" system that updates rules in minutes, dramatically increasing the proportion of historical defects covered by new test cases (from 12% to 76%).

These numbers demonstrate that the improvement was not simply AI becoming smarter, but the accumulation of domain knowledge and clearer quality‑decision logic behind the system.

Interviews with test engineers reveal a common misconception: AI will replace testers. In reality, the work of testers is changing. Tasks that remain critical and cannot be fully automated include:

Diagnosing why AI‑generated scripts fail in real environments.

Locating intermittent failures that cannot be blamed solely on backend instability.

Handling environment differences such as permissions, caches, and gray‑release states.

Deciding when to trust automation results and when human intervention is required.

These activities, formerly called "test automation maintenance," are now framed as "harness engineering"—designing and troubleshooting the quality system itself.

The core transformation is a shift from a linear testing workflow (PRD → design cases → manual execution/script → run → record results) to a systemic role where testers define what constitutes valid verification and ensure the system can understand and enforce that definition.

Key lessons from the Kuaishou case:

Upstream quality strategy: Move from "give me a requirement, I write cases" to identifying real risks, hidden business rules, and recurring pitfalls.

Systematic knowledge: Convert raw defect records into structured knowledge assets (templates, rule graphs) that AI can query.

Execution credibility: Focus on stable, evidence‑rich test runs rather than sheer coverage numbers.

Decision weight: When hundreds of auto‑generated cases are presented, prioritize the few that truly matter and understand the impact of a 1% failure.

For individual testers, the article proposes a four‑step roadmap:

From "how to verify" to "what to verify": Analyze historical defects in your module to identify the most critical scenarios.

Build a knowledge asset: Organize defects into a hierarchy (category → trigger scenario → root cause → verification method → reproducible steps) so AI can reuse it.

Prioritize stability and evidence chains: Prefer a smaller set of tests with clear evidence over a large set that merely appears to pass.

Master tool harnessing: When AI‑generated scripts fail, investigate the underlying strategy, adjust constraints, enrich context, and refine the review process to turn the tool into a reliable partner.

Ultimately, the article argues that testing is not being eliminated by AI; instead, the definition of testing is evolving. Test engineers must transition from merely executing verification steps to becoming architects of the quality system, defining correct validation, guiding AI with domain knowledge, and maintaining the decision logic that underpins trustworthy automation.