How Tmall’s AI Transforms Test Case Generation for Faster, Smarter QA

Background

With the rapid evolution of large models, the testing industry is exploring AI‑based solutions, typically using prompt + RAG to build agents for requirement analysis, test case generation, and data construction. Various industry approaches and results are presented using test case generation as an example.

Industry Characteristics

In the fast‑changing e‑commerce sector, rapid iteration and high quality demand pose challenges for testing teams, including fast version cycles, high labor costs, and bottlenecks in traditional test case design, maintenance, and execution.

Fast version rhythm, high labor cost: Rapid releases require extensive testing effort while controlling personnel expenses.

Traditional testing bottlenecks: Designing, maintaining, and executing test cases—especially for complex scenarios—relies heavily on engineers' experience and time.

Additional pain points in test case authoring include low design efficiency, requirement misunderstanding, insufficient knowledge consolidation, and high manual cost due to repetitive work.

The business is divided into five types—marketing solutions, guide‑shop domain, transaction settlement, multi‑department collaboration, and back‑office—each requiring tailored case generation.

Implementation Strategy

The core goal is to use AI to automate test case generation and build an industry‑specific testing flow.

Case Generation Approach

The QA workflow is visualized as: requirement understanding → risk assessment → test case design → execution → defect tracking → integration/regression → release → feedback.

Since test case design consumes about 70% of QA time, AI assistance is urgently needed.

Specific Implementation Plan

Prompt Engineering & Process Optimization: Refine prompts with context to guide LLMs in generating high‑quality cases and integrate the end‑to‑end flow into existing workflows.

High‑Quality Knowledge Base Construction: Systematically capture business background, baseline cases, edge cases, and loss‑scenario data, then apply RAG to improve relevance and accuracy.

Requirement Normalization: Standardize PRDs with templates to enhance input quality and stabilize AI‑generated case coverage.

AI Agent Empowerment: Develop agents for knowledge‑base building, PRD completion, and knowledge checks, reducing manual effort.

Platform Integration: Embed AI capabilities into the test‑case management platform for visual operation, supporting both AI‑Test and Test Copilot modes.

Knowledge Base Details

Scope: Test cases (baseline, edge), business background (terms, processes, user stories), loss‑scenario details.

Data Format: Structured storage using plain text, markdown, JSON, and tables.

Retrieval: Chunking, recall strategies, and indexing methods.

Maintenance: Data cleaning, deduplication, and update mechanisms.

Application Effects

Adoption Rate: Over 85% in C‑end scenarios (e.g., guide, detail pages) but below 40% in B‑end areas (e.g., finance, supply chain).

Efficiency Gains: For small‑to‑medium requirements in guide and marketing domains, case authoring time dropped from 2 hours to 0.5 hours, saving 75% of effort.

Outlook

Future work aims to improve PRD quality, support visual and interaction drafts, and handle complex requirements more effectively, moving toward fully automated AI‑driven testing pipelines that cover requirement analysis, case generation, script construction, execution, defect reporting, and quality closure.