How Large Language Models Can Transform Software Testing

1. Automated Test Case Generation

Requirements specifications and design documents contain detailed functional descriptions. Large models can understand these texts, extract key information such as inputs, expected outputs, and boundary values, and generate initial test cases covering common scenarios and basic exceptions. Human review is still needed to ensure accuracy.

2. Defect Prediction

Historical defect data (type, location, trigger conditions) can be learned by large models to discover patterns. Combined with static and dynamic code analysis, the model can identify risky code regions and predict potential defects by matching current code structures with past defect signatures.

3. Test Result Analysis

Massive test execution data—including pass/fail status, timing, and logs—can be aggregated and mined by large models. Machine‑learning algorithms reveal hidden patterns, such as operation sequences that cause crashes or modules with rising error rates, helping testers locate and resolve issues faster.

4. Optimizing Test Strategy

Based on overall software understanding (feature importance, complexity, change frequency), the model suggests testing priorities, allocates more resources to critical or volatile functions, and highlights uncovered code areas to improve test coverage.

5. Intelligent Test Execution

During automated runs, the model monitors progress and outcomes. If a step repeatedly fails, it analyzes causes and may adjust subsequent steps or parameters (e.g., increase concurrent users for performance tests) to explore limits and improve efficiency.

6. Compatibility Testing Assistance

By leveraging past compatibility data and platform characteristics, the model can forecast potential issues on new OS or hardware combinations, prompting targeted testing to mitigate risks.

Conclusion

While large models greatly enhance testing efficiency, they cannot fully replace human expertise; testers’ experience and domain knowledge remain essential, and the models should be used as supportive tools.

Large Model Tools

OpenAI GPT‑3 or GPT‑4

Google Vertex AI

Baidu Wenxin Yiyan

Case Studies

Enterprise management software: GPT‑4 generated comprehensive test cases, improving coverage.

Social media app: GPT‑4 suggested user interaction paths, uncovering early UX defects.

Financial transaction system: GPT‑4 analyzed performance data, identified bottlenecks and optimization suggestions.

Game development: GPT‑4 created diverse storyline and mission test scenarios.

Mobile app compatibility: GPT‑4 predicted device‑OS issues, enabling focused testing.

GPT‑4 Test Case Generation Process

Preparation

Define test goals, scope, and priorities.

Gather requirements, design docs, user manuals, and past defect reports.

Plan test types (functional, performance, compatibility) and priorities.

Interaction with GPT‑4

Select an API or integrated tool.

Provide clear software description, functional flow, and example scenarios.

Specify the desired test case format and coverage.

Generation and Evaluation

Review GPT‑4 output for completeness and executability.

Compare with existing test ideas and refine.

Optimization and Finalization

Give feedback to GPT‑4 for improvements.

Manually adjust and supplement test cases (boundary, stress, etc.).

Conduct peer review and document finalized cases.

Execution and Maintenance

Run tests, record results, and feed new issues back to GPT‑4.

Periodically update test cases as software evolves.

Example: E‑commerce Order & Payment Test Cases

Normal Flow

User adds a product to cart, proceeds to checkout, selects WeChat Pay, enters correct password, and completes payment; verify success message, order status, and notification.

Multiple items with Alipay, correct address, and successful payment; verify accurate amount, discounts, and order status.

Discounted item with bank card payment; verify discount applied and order updates.

Exception Scenarios

Incomplete address triggers error, preventing payment.

Switching payment method after cancellation works correctly.

Wrong password three times locks account; payment fails.

Network interruption during payment either resumes or cancels appropriately.

Insufficient stock blocks checkout.