How an AI‑Powered Defect Analyzer Eliminates Confirmation Bias in Bug Investigation

Problem addressed

Test engineers often form a hypothesis from a single clue, which can lead to costly misdiagnoses such as adding indexes for a perceived slow query that does not actually fix the issue.

11‑part analysis framework

1. Defect Phenomenon

Operation : User selects a coupon on the order page and clicks “Submit Order”.

Expectation : Order completes within 2 seconds and redirects to the payment page.

Actual : Spinner lasts >10 seconds, “system busy” is shown, and no order is generated.

2. Impact Scope

Severity : P0 – core ordering function broken.

Frequency : High‑frequency intermittent (started at 10:00 am, multiple user reports, not 100 % reproducible).

3. Known Facts

Page spinner >10 s – evidence: user screenshot + support feedback.

Final error “system busy” – evidence: user screenshot.

Order works without coupon – evidence: partial user feedback.

Problem started at 10:00 am – evidence: support timeline.

4. Unknown Information

Exact reproducible path (all coupons or specific ones?)

Error logs and response codes

Affected version and whether yesterday was normal

User‑level patterns (device, coupon type)

5. Possible Causes

Coupon discount service slow query / deadlock – Likelihood : High – Evidence : normal without coupon → issue appears only with coupon; 10 s spinner suggests DB lock wait.

Coupon service timeout without degradation – Likelihood : High – Evidence : final “system busy” indicates possible timeout with circuit‑breaker not triggered.

Coupon status validation bug – Likelihood : Medium – Evidence : order not generated → possible infinite loop in checkCouponStatus().

Database connection pool exhaustion – Likelihood : Medium – Evidence : sudden appearance at 10:00 → possible batch job consuming connections.

Frontend parameter error – Likelihood : Low – Evidence : theoretically possible, no direct evidence.

6. Verification Methods

Slow query / deadlock : Search slow‑query logs after 10:00 for SQL containing “coupon”. Expected result: execution time >10 s or lock‑wait records. Cost: Low.

Service timeout without degradation : Inspect the call‑chain of the “submit order” request, check coupon‑service latency and circuit‑breaker status. Expected result: timeout observed + circuit breaker not triggered. Cost: Medium.

Status‑validation bug : Reproduce in test environment with debugger, watch whether execution stalls in checkCouponStatus(). Expected result: thread stuck. Cost: Medium.

7. Recommended Investigation Order

Check slow‑query logs and connection‑pool metrics (≈5 min) – validates causes 1 and 4.

Review call‑chain monitoring (≈10 min) – validates cause 2.

Add logs in test environment to reproduce (≈30 min) – validates causes 1/2/3.

Ask operations to review change records around 10:00 (meeting may be needed).

Sorting principle: “do the cheap things first” – high‑likelihood & low‑cost items first.

8. Information Needed When Evidence Is Insufficient

Error logs : Full logs of the order‑submission API after 10:00 am – acquire from ELK.

API response time : P50 / P99 distribution – acquire from APM monitoring.

Call chain : Trace ID of failed requests – acquire from tracing platform.

Slow queries : SQL >1 s after 10:00 am – acquire from DBA or DB monitoring.

Change records : Code/config changes in the 24 h before 10:00 – acquire from release platform.

9. Possible Fix Directions (after root cause confirmed)

Optimize slow query : Add index / reduce JOIN – eliminates root cause but may affect other features.

Configure circuit‑breaker degradation : Set timeout 2 s, bypass coupon check on timeout – provides quick stop‑bleed but disables coupon validation (requires compensation).

Temporarily disable coupons : Deploy config to hide coupon entry – effective within 5 min but degrades user experience.

10. Post‑Fix Validation Recommendations

Functional verification : normal flow + error flow + boundary values.

Regression verification : full coupon test suite + full order‑process suite.

Data verification : ensure no “order created but coupon not deducted” dirty data.

Monitoring verification : P99 response time <2 s.

Long‑term observation : watch 24 h for new “system busy” incidents.

11. Regression Test Points to Prevent Recurrence

Concurrent case : Same coupon used concurrently in two orders, only one succeeds – priority P0.

Exception recovery : Coupon service timeout → circuit‑breaker works, order not blocked – priority P0.

Boundary value : Coupon value equals order amount – priority P0.

Performance : Coupon‑discount API P99 <500 ms – priority P1.

Analysis quality red lines

Do not draw conclusions without evidence.

Do not write assumptions as facts.

Do not list only one possible cause.

Do not leave verification method vague (e.g., “check it”).

Do not write generic fix without specifics.

Do not ignore unknown information.

Do not omit investigation steps.

Do not leave impact scope un‑rated.

Do not omit likelihood rating for causes.

Do not forget to list missing information when evidence is insufficient.

Forced behaviours

Separate facts and assumptions (sections 3 vs 5).

Every possible cause must have a verification method.

Prioritise investigation by “likelihood × cost”.

Impact scope must include a severity rating.

If information is lacking, ask for it first instead of forcing analysis.

Special scenario requirements (selected examples)

Online P0 incidents : include screenshots, user‑count estimate, temporary degradation plan, fault‑notification template.

Performance issues : require slow‑query logs, P99 from APM, GC frequency; possible causes include missing index, cache miss, Full GC, deadlock; verification must include EXPLAIN execution plan.

Data‑consistency problems : record earliest detection time and affected rows; possible causes include concurrency, uncommitted transaction, async‑task failure, missing idempotency; evaluate need for data‑repair scripts.

How to use the skill

Method 1: WorkBuddy skill

Place the defect-analyzer-cross-platform folder into ~/.workbuddy/skills/ and invoke “Analyze this defect” in a conversation.

Method 2: Copy prompt

Open prompt-template.md, copy its entire content, and paste it into any LLM interface (e.g., Cursor, Claude, ChatGPT, DeepSeek, Kimi) followed by the bug description.

Method 3: Python script

# Generate the 11‑part analysis framework from a defect description
python3 generator.py --input defect_description.md --output analysis_framework.md

# Check an existing draft for missing parts
python3 generator.py --input draft.md --check-only

Method 4: Cursor / Claude project rules

Store prompt-template.md in .cursorrules (for Cursor) or CLAUDE.md (for Claude); all AI conversations in the project will automatically follow the 11‑part structure.

Conclusion

Without a structured analysis report, the original incident cost tens of thousands of yuan in wasted effort. By separating facts from assumptions, enumerating multiple hypotheses, ranking them by likelihood and cost, and defining concrete verification steps, the defect‑analyzer skill guides engineers to faster, more reliable root‑cause discovery.