AI-Era Quality Shift-Left: JD Health's 6-Phase AI Agent System Cuts Production Bugs 65%

The Cost of Late Detection: A 100x Price Gap

In January 2019, a Pinduoduo operations team was awakened to discover a "100 yuan no-threshold coupon" that could be claimed infinitely across the platform. Within minutes, bargain hunters exploited it, causing tens of millions in losses. Post-mortem labeled it a "technical vulnerability," but the root cause was not a code error — it was a complete absence of business rule clarity at the requirements stage: no boundary definition for the coupon, no frequency limit, no risk-control fallback.

JD Health SRE lead and technical committee member Li Xun, a veteran of 618 and Double 11 full-chain guarantees, presented striking data at the 10th AI+R&D Digital Summit: 80% of online issues originate in the requirements phase , and fixing a defect at requirements costs over 100x less than fixing it after release.

Rocket vs. Bicycle: The AI Imbalance

Development has accelerated dramatically: code output up 10-15x, PR submissions up 3-5x, delivery cycles compressed from weeks to days. GitHub Copilot boosts coding speed by 55%, with enterprise adoption exceeding 60%. Meanwhile, testing headcount remains flat, analysis and execution are still manual, and test windows are squeezed further. Testing has become the hardest bottleneck — "the only solution is four words: use AI to counter AI."

But simply using AI to "test faster" is just putting an electric motor on a bicycle. The real problem: many risks are baked in at the requirements stage , which traditional quality systems completely miss.

Case Study: One Reward Requirement, Five Hidden Risks

A routine marketing reward requirement passed traditional review ("looks fine") but AI review uncovered five defects:

Missing reward condition boundaries: Requirement stated "user participates to get reward" but didn't define multi-device participation. Result: duplicate rewards, direct financial loss.

Document logic inconsistency: PRD rules differed from UI display. Users followed UI, system followed PRD — complaints and rework.

Collaboration risk: Upstream/downstream interface fields changed silently; requirement doc not updated. Discovered only at integration, blocking the whole chain.

Missing exception fallback: No handling for reward timeout or failure. Production timeout triggered user complaints and ops alerts simultaneously.

Idempotency logic undefined: Repeated clicks on "claim" — would system deduct repeatedly? Requirement silent, code implemented naively — duplicate deductions, more financial loss.

All five were not code bugs but requirement omissions . Fixing them at requirements stage costs near zero (edit docs, add rules); letting them reach production means financial loss, complaints, alerts, all-night post-mortems, and lost user trust.

Invisible Risks Are the Deadliest: The Therac-25 Lesson

1985-1987: Canada's Therac-25 radiation therapy machine delivered 100x overdoses due to a race condition, killing three patients. Investigation revealed: (1) hardware safety interlocks removed, safety entrusted entirely to software; (2) software reused from prior model without full requirements validation. The race condition did not exist in requirements because no one defined "what if user operates UI at extreme speed?"

Testing can never cover a scenario that doesn't exist in requirements. You can invest 10x test effort, but the test ceiling is set at requirements. This drives the paradigm shift Li Xun emphasizes: quality moves from "a gate in testing" to "an automated pipeline across the entire R&D flow" — from post-hoc testing to full-chain left-shift with requirements-phase intervention; from single-modality (text/code) to multi-modal (PRD+UI+code+logs+monitoring+feedback); from human decisions to AI autonomous loops + human-AI collaboration; from generic LLMs to quality-domain vertical models.

Method Over Tool: Six-Phase Closed Loop with Ten AI Agents

Li Xun warns: point tools cannot solve systemic problems. A risk flows through "identify → grade → test strategy → execute → report → close." If a high-risk item identified at source isn't graded, doesn't drive test strategy, isn't tracked to closure, the identification is wasted.

JD Health built a six-phase closed-loop system powered by ten specialized AI agents sharing a knowledge base and data layer:

Requirements Phase: Document Parsing Agent + Requirements Review & Risk Identification Agent + Security Control Agent. Auto-parse PRD semantics, detect ambiguities, logical contradictions, compliance issues; output multi-dimensional scores (completeness, consistency, testability, compliance); security governance (data desensitization, privacy compliance, vulnerability risk); change impact analysis via call-chain analysis to eliminate "changed one line, don't know business impact" blind spots.

Test Case Design Phase: Test Planning Agent + Test Case Generation & Review Agent. NLP parses requirements to auto-generate structured test cases covering equivalence classes, boundary values, exception scenarios, combinatorial scenarios; simultaneously produces executable UI automation scripts. Test design time compressed from days to minutes.

Pre-Test Phase: Code Review Agent. Combines semantic understanding and context-aware static analysis + security scanning; high-risk defects auto-block MR merge; provides actionable fix suggestions for specific code, not generic templates.

Test Execution Phase: Automation Execution Agent (multi-end script scheduling, self-healing), White-box Test Agent (coverage-driven generation of uncovered branch cases), Defect Analysis Agent (auto-pulls defect data, trend alerting).

Release Readiness Report Phase: Test Report Agent aggregates full-flow data, gives go/no-go recommendation: pass, conditional pass, or fail.

End-to-End Risk Tracking: Risk Tracking Agent manages full lifecycle of every risk: identify, grade (P0-P3), dispose, verify, close, archive — ensuring every risk is "seen, tracked, closed."

Two key design highlights:

AI Acceptance Gate: Intelligent gate at each phase entry uses risk level, coverage, pass rate, defect trends to auto-decide pass/conditional pass/fail. Fail blocks release, triggers repair loop. "AI gate doesn't replace human judgment; it ensures every human decision is based on more complete information."

Human-AI Collaboration & Dual Deployment: AI handles repetitive judgment, data aggregation, risk warning, fix suggestions; humans retain key decisions — release approval, knowledge audit, strategy tuning. Cloud agents run full-loop; local agents run in developer IDEs for instant Q&A, unit tests; sensitive code stays local, only desensitized results reported. Clear boundaries, each plays its role.

AI Needs Feeding: Model Is Just the Starting Point

Out-of-the-box LLMs are like brilliant interns who know nothing of your business. They can spot generic boundaries and idempotency, but won't know "JD Health's drug subsidy rules" or "medical compliance red lines" unless fed business knowledge.

Three feeding paths:

Feed Business Knowledge: Thousands of business rules and historical failure cases built into a knowledge base; RAG lets model "review with memory."

Feed Rule Engineering: Knowledge graphs structure system dependencies; rule engines turn compliance into hard constraints; dynamic prompt injection (general + business-specific + user-role three layers), few-shot examples, and chain-of-thought reasoning steer model onto correct track.

Feed Continuous Feedback: Every production escape auto-triggers retrospective: why wasn't it caught earlier? Converted into new risk rule, backfilled into knowledge base. Next occurrence auto-intercepted. Like immune system: one loss, one antibody. Result: risk recall rate >85%, miss rate on par with human experts — credit goes to feeding, not the model alone.

Challenges: business variance handled by configurable platform; hallucinations mitigated by structured context constraints and human-in-the-loop review; hardest is trust — so every output is explainable, traceable, human-correctable. AI suggests, human overrides, system learns from correction.

Numbers Tell the Story

Requirements assessment efficiency: from 1-3 hours manual to minutes, >90% improvement.

Test design speedup: single requirement from 180 minutes to 90 minutes, 50% reduction.

Defect early interception: requirements-phase interception rate >70% — seven out of ten issues blocked at source.

Production escape rate: overall reduction of 65%.

Coverage: 10+ C2-level business teams, 20+ business lines, 300+ requirements assessed monthly, 1,200+ domain knowledge items accumulated. Economics: requirements-phase doc change costs 1 unit; production fix costs 100-1000 units. Enterprises with thousands of monthly requirements see escape rates, complaints, rework costs all bend downward within 3-6 months of front-loading quality investment.

Shift-Left Redistributes Responsibility Across the R&D Chain

Product Managers: Requirements must be "clearly stated." Boundaries, exceptions, idempotency, fallbacks become first-class citizens in docs, not holes developers patch later. Vague requirements — AI can't save them.

Developers: Faster coding ≠ skip quality thinking. AI catches null pointers and injections, but business semantics and upstream/downstream impact still need your judgment. Every "small change" deserves: "Who does this affect?"

Test Engineers: Role upgrades, value rises. No longer just gatekeepers at finish line; become scouts at source: join requirements reviews, define risk-driven test strategies, feed knowledge bases. AI takes repetitive work, leaves hardest decisions to you.

Managers: Quality responsibility shifts from "test department KPI" to "full-chain shared contract." Investment needed not just in tools but in knowledge accumulation and process redesign. AI without knowledge base is a fancy toy; AI without process support is another silo.

Conclusion

Back to that 2019 night. If the coupon requirement had been asked at review: "What are boundaries? Idempotency? Fallback?" — that night's disaster would never have happened.

In the AI era, upfront quality work is no longer a multiple-choice question but a mandatory one. The development rocket has launched; quality waiting at the finish line will be smashed. The critical second step: shift-left must have method. Not throwing an LLM at a review meeting, but building the complete "identify → grade → strategy → execute → report → close" system, giving AI business knowledge to rely on, rules to follow, feedback to learn from.

Li Xun's closing vision: build a "can understand, can predict, can self-evolve" next-generation intelligent quality control system — from "assist decision" to "autonomous execute + human review," from generic LLM to vertical quality LLM. The road is long, but the direction is crystal clear.

Quality is not tested in; it is built in from the source. Building a dam at the source beats salvaging ten thousand times downstream. True in the AI era, true in any era.