Why More Test Cases Can Increase Bugs and How This AI Skill Closes Testing Gaps

Test Cases vs. Test Points

Many teams confuse test cases (the "how to test" steps) with test points (the "what to test" coverage checklist). Test cases describe actions, inputs, and expected results, while test points define the scope of what should be verified, such as discount‑coupon precision, boundary values, and concurrency.

The Need for Structured Coverage

In a real incident, a P1 bug occurred when a discount coupon reduced the order amount to a negative value. The post‑mortem revealed that out of 186 test cases, fewer than 30 covered amount‑related scenarios, and none addressed the "discount‑coupon rounding chain when the order amount is a decimal". The root cause was not a shortage of test cases but an incomplete set of test points.

Introducing test-point-generator

The test-point-generator skill automates the "brain‑storming" step by converting it into a structured, repeatable process. It takes a requirement document, scans it against a fixed 14‑dimension checklist, and outputs a list of test points.

14 Dimensions

The dimensions are derived from categorizing past production bugs and missed test cases. They include:

D01 Normal Flow – Is the core path functional?

D02 Exception Flow – How does the system react to wrong input, network failure, or downstream errors?

D03 Boundary Values – Max, min, edge, and null values.

D04 Permission – Access control and privilege escalation.

D05 Data Validation – Front‑end vs. back‑end checks, required fields.

D06 State Transition – Illegal state moves or unreachable states.

D07 Concurrency – Simultaneous operations on the same resource.

D08 Idempotency – Duplicate submissions.

D09 API – Missing parameters, type errors, pagination limits.

D10 Database – Write correctness and transaction rollback.

D11 Logging/Audit – Critical actions recorded and sensitive data masked.

D12 Compatibility – Browser, device, and platform variations.

D13 Performance – Response time, resource usage, long‑run stability.

D14 Regression – New features must not break existing ones.

Running through all 14 dimensions provides a checklist‑style scan that dramatically reduces the chance of missing a test point.

Special Scenarios with Mandatory Rules

Two categories require extra coverage:

Monetary Scenarios – Precision, boundaries (zero, negative, budget exhausted), combination rules, cumulative deductions, and refund rollbacks. Whenever keywords like "amount", "discount", "full‑reduction", or "payment" appear, the corresponding test points are forced and must constitute at least 30% of the dimension’s total.

AI Interaction Scenarios – Intent understanding, ambiguity clarification, context consistency, refusal handling, hallucination control, and security (prompt injection, data leakage). These differ from traditional deterministic tests because the same input can yield different outputs.

Output Formats

The tool supports two formats:

Table format – Ten fields per test point (ID, module, scenario type, description, related rule, test data, expected result, priority, automation suitability, risk). The table can be copied directly into Excel or Feishu.

Mind‑map format – A Markdown outline that can be imported into Feishu mind‑map or XMind, grouped by dimension.

Red‑Line Rules (What Not to Write)

Do not write step‑by‑step actions; that belongs to the downstream testcase-writer skill.

Do not invent product rules; mark them as "to be confirmed" or "AI inference".

Every test point must have a concrete, assertable expected result (e.g., HTTP 400 with error code INVALID_AMOUNT).

Avoid vague statements; specify exact values and conditions.

For monetary and AI scenarios, ensure the number of test points is at least 30% of the total for that dimension.

Real‑World Demo: E‑commerce Coupon

A full‑reduction coupon ("spend 100 get 10 off") is used as a demo. The skill performs three preprocessing steps:

Identify business rules (e.g., BR‑01 to BR‑07).

Identify boundary conditions (amount threshold, validity period, per‑user limit).

Identify pending questions (refund handling, validity calculation).

Scanning the 14 dimensions yields 67 test points, including examples such as:

TP-013: Remaining stock = 1, claim succeeds, stock becomes zero, status = "claimed" (P0)
TP-015: Use on the 7th day 23:59:59, expect success (P0)
TP-016: Use on the 8th day 00:00:01, expect failure with "expired" (P0)
TP-019: Same user clicks claim 5 times within 1 s, only 1 succeeds (P0)
TP-035: Two concurrent claims, only one succeeds (P0)
TP-048: Discount 1% on order 0.01, expect payment 0.00 (P0)
TP-049: 9.9% discount on large order, expect correct deduction (P0)

A separate list of 16 pending questions (6 of them P0) is also generated.

Toolchain Position

The skill sits as the fourth ring in the testing toolchain:

Upstream: requirement-decomposition – splits requirements, does not generate test points.

Current: test-point-generator – produces test points only.

Downstream: testcase-writer – expands test points into full test cases.

This separation allows each skill to be upgraded or replaced independently.

Four Usage Modes

Directly in WorkBuddy: paste the requirement and ask for test points, optionally specifying "mind‑map" output.

Other LLMs (Claude, ChatGPT, etc.): copy the prompt-template.md and append the requirement.

Command‑line:

# Table format (default)
python3 generator.py --input requirement.md --output test-points.md
# Mind‑map format
python3 generator.py --input requirement.md --output test-points-mindmap.md --format mindmap

Browser bookmarklet: add bookmarklet.html to the toolbar to launch the AI dialog with a pre‑filled prompt.

The tool can be integrated into CI pipelines to automatically re‑run on each PRD change.

Pitfalls Encountered

Vague expectations – Early versions produced generic results like "system should handle correctly"; adding the red‑line rule forced concrete, assertable expectations.

AI‑inferred rules – When requirements omitted refund logic, the AI guessed industry conventions, leading to incorrect test points. Marking all AI inferences as "AI inference" solved the issue.

Missing special scenarios – Monetary precision and AI safety were initially omitted; explicit keyword triggers and mandatory coverage percentages were added.

Conclusion

If the 14‑dimension scan had been applied before writing the original test cases, the discount‑coupon rounding issue would have been caught, preventing the P1 incident. The skill’s value lies in systematic, consistent coverage that narrows the gap between junior and senior testers.

Although the skill itself is simple, solidifying the "brain‑storm" step into a repeatable checklist can close many hidden bugs.