Amap Marketing's AI Quality Engineering: Closing the Risk-to-Verification Loop
Amap Marketing implements AI quality engineering with Graph Engineering decision graphs for risk identification and AgentLoop workflows for test verification, linked by a unified evidence protocol. The practice uses LangGraph for multi-role code review, Safe Autofix with permission ladders, a four-stage AgentLoop addressing the oracle problem, self-healing constraints, and a case study showing risk-to-fix traceability.
Background
Amap Marketing defines AI quality engineering as integrating AI analysis and execution into software quality assurance: AI assists in risk identification, test design, and verification, constrained by explicit business rules, permission boundaries, human review, and execution logs. The core challenge is connecting risk evidence, test expectations, and actual results into a traceable delivery basis.
The team built two systems: Graph Engineering decision graphs on the R&D side for multi-role risk review, and AgentLoop workflows on the testing side for change analysis, test case design, test planning, and automation. A unified evidence protocol (still in progress) aims to link both sides.
R&D Side: Graph Engineering Decision Graph
Verifiable Risk Conclusions via LangGraph StateGraph
The review process is explicitly orchestrated with LangGraph StateGraph: Generator creates initial Finding items from code changes.
Roles PM, QA, Developer, SRE, Security review in parallel — covering requirement boundaries, test conditions, stability, and security. Reducer merges duplicate conclusions. Judge assigns validity, severity, and priority.
When policy allows, Executor / Refiner produce fix proposals and iterate based on verification diagnostics (limited iterations).
Each output must state the risk, its code location, and the verification conditions needed to confirm mitigation. For example, "new enum missing fallback branch" must include change location, affected business conditions, and test scenarios to become usable by testing.
Safe Autofix Permission Ladder
Risk judgment and execution permissions are separated. Safe Autofix splits the fix flow into policy-controlled steps:
proposal → dry-run → apply → verification → commit → push → optional MRBefore fixing, the system checks policy whitelist, risk level, confidence, and evidence anchors. A dry-run executes before actual changes; repository verification runs after. commit, push, and MR are each gated — passing verification does not automatically grant further delivery rights. Every attempt logs verification results, failure reasons, and action source.
On verification failure, command output and diagnostics feed back to Refiner. After iteration or budget limits, auto-attempts stop; local changes are handled per execution state and turned into report items; remote actions need separate handling. Current snapshots show Autofix records at 0 — the focus remains on risk identification, human confirmation, and boundary validation.
Testing Side: AgentLoop Verification Closure
Oracle Problem: Sourcing Expected Results
Generating compilable test code does not guarantee correct business behavior. If an agent derives expected values from implementation code, implementation errors may be copied into assertions. The solution: annotate expectation sources.
Requirement documents provide implementation-independent rules.
Commit messages assist intent understanding but require cross-check.
Expectations derived solely from code can only describe code behavior, not prove business correctness.
Phase 1 extracts requirement_expectations[] — rules with applicability conditions — passed to later phases. When independent rules are missing, verification capability is explicitly marked as limited rather than promoting code behavior to business standard.
For critical risks, actual response, requirement rules, and test assertions are compared. Scenarios are designed to distinguish target behavior from erroneous behavior (e.g., legal vs. illegal inputs must yield different assertions). Positive/negative checks improve discrimination, but passing limited cases does not prove all paths correct; requirement accuracy still needs human judgment.
Four-Stage AgentLoop
AgentLoop produces test assets through four core stages, then executes verification, failure diagnosis, coverage analysis, and reporting.
Core Stages and Artifacts
① Change Analysis : Identify change scope, risk points, requirement expectations → change-impact-analysis.md ② Test Case Design : Determine business scenarios, inputs, assertions → test-cases.md; check constructibility, duplication, boundary realism, observability, environment feasibility.
③ Test Plan Design : Map scenarios to test framework, data prep, execution → test-plan.md.
④ Automation Implementation : Generate test code and implementation summary; combine execution results to locate failures, produce auditable verification records.
Human review gates separate adjacent core stages. Stage 4 allows limited in-stage revision; if root cause lies in upstream business understanding or test design, the flow pauses for human confirmation — avoiding mixing different concern layers in one generation. Kbase supplements environment knowledge missing from code diffs (cache rules, Mock strategies, TestBase differences). Structured entries feed into case design, plan design, and implementation, syncing with knowledge base and repo updates. It provides context for environment adaptation but does not replace requirement rules as constraints on test expectations.
Self-Healing Rules & Defect Classification
On test failure, the system first classifies: environment flakiness, data residue, test case issue, expectation error, or service implementation defect. Attribution relies on request, response, environment state, and independent requirement evidence; insufficient evidence yields "pending judgment" — never reverse-engineer passing criteria.
Self-healing obeys three constraints:
Expectation revisions require independent evidence — cannot copy actual response.
Cannot delete assertions or lower precision to hide failures.
All revisions are recorded.
Cases revised three times are flagged as weak verification for human audit. Revision count does not determine correctness; verification conclusion still depends on independent expectations, execution results, and coverage. Upstream rule or design changes require returning to human confirmation.
Case Study: From Risk Identification to Re-test
Current R&D-test handoff is manual; unified protocol, auto-handoff, and result backflow are under construction. A marketing change adding industry-type enum and static voucher face value illustrates the flow.
R&D Identifies Fallback Risk & Verification Demand
The change introduced new industry types. R&D's Generator produced a Finding: new type missing fallback path. Multi-role review noted this logic links to voucher face value — omission could affect fund-related behavior — so Judge demanded verification.
This Finding focused testing on a concrete question: does industry parameter change cause static voucher face value to deviate from business rules? R&D specifies suspicious code paths; testing defines expected results for legal/illegal inputs and checks consistency via execution.
Testing Transforms Risk into Seven Test Cases
Testing followed the four stages. Phase 1 extracted industry parameter ↔ face value expectations from requirements. Phase 2 expanded into seven cases; TC-03 specifically checked differential behavior between legal and illegal industry inputs. Phase 3 mapped to data prep, API calls, execution conditions. Phase 4 implemented automation and ran tests.
The test design asserted: legal industry parameter returns configured face value; illegal returns fallback value. TC-03 verified this distinction. Note: requirement wording on rule scope had ambiguities; this case illustrates differential handling, not full business validation.
TC-03 Finds Return Difference → Fix → Re-test
First TC-03 run: illegal industry input returned configured face value, contradicting the fallback expectation. The failure concretized the earlier code risk into specific input, expectation, and actual result, focusing diagnosis on the illegal-input handling path.
Developer located missing fallback branch and added handling. TC-03 re-test passed; the Finding was marked "verified fix". This linked R&D-identified risk, test-execution delta, code change, and re-test result. The reported verification is limited to TC-03 's handling record.
Execution and Handling Records for TC-03
Fallback handling : First execution — Developer identified missing fallback branch. Subsequent handling — Developer added corresponding handling.
TC-03 execution : First execution — Illegal industry input returned configured face value, differing from case expectation. Subsequent handling — Case record shows re-test passed.
Finding status : First execution — R&D raised fallback risk needing verification. Subsequent handling — Case record marked "verified fix".
This record covers only TC-03 's first execution and follow-up, not the other six cases, nor full request/response/assertion details. It demonstrates how one risk gets located and re-tested, not overall business coverage. Complete delivery evidence should also link independent business rules, pre/post-fix versions, and corresponding execution results.
Linking Risk & Verification as Delivery Evidence
The case requires correlating concrete items: the enum change, the fallback Finding, TC-03 inputs/expectations, first-run delta, developer's branch fix, re-test result, and Finding status. Placing these in one record enables tracing from "what risk was found" to "what was changed and what result judged it effective".
Planned evidence model uses minimal core fields: change_id + source_revision — identify change and version. claim / locator — describe risk or verification conclusion and its basis. verification_status — distinguish pending, human-confirmed, execution-verified states.
Consistent identifiers linking requirement basis, code changes, and run results make delivery judgments auditable. The case exemplifies the collaboration mode: R&D-discovered risks enter concrete test scenarios; execution results enter issue disposition records. Every delivery judgment must state its basis, version, and verification scope to enable traceback to specific inputs and results when disputes arise.
Phase Snapshots & Applicability Boundaries
Observing Issue Disposition & Workflow Completion via Dashboards
R&D and testing dashboards track issue handling and test work-item flow states separately. Numbers below are from single dashboard screenshots, limited to displayed record sets. The two boards are not the same statistical queue; they are explained separately, not used to calculate end-to-end delivery effects or long-term efficiency trends.
Dashboard Metrics
Business Repo A Issue Handling : 94 issue records, 50 marked trusted fixes; 50 ÷ 94 = 53.2%. Scope: the 94 records in this snapshot. Board also has reported fixes and Autofix categories, not merged into auto-fix effectiveness.
Test Work-item Flow State : 46 work items, 34 marked full-stage complete; 34 ÷ 46 ≈ 74%. Remaining 12 incomplete. Scope: the 46 work items in this snapshot. Incomplete is a flow state, not equal to defects or test failures.
Dashboard shows 53.2% trusted fixes (60 reported fixes include 10 line-drift misjudgments); Autofix = 0, two fix record calibers differ. Test work items across Jun/Jul/Aug labels: 13, 19, 14 items (total 46); completion ratio describes flow state of those items. "Full-chain delivery" refers to this test workflow, not cross-R&D-test auto-delivery.
First dataset shows single-repo issue handling state with Autofix at 0; second shows displayed work-items' flow completion. Flow completion ≠ business verification pass; single snapshot ≠ overall quality or efficiency trend.
Applicability & Next Steps
This practice suits scenarios with clear business rules, controllable environments, and observable results. Incomplete requirements, complex concurrency, or probabilistic outcomes still need human judgment and other verification methods. Both sides' capabilities are landed; unified protocol, auto-handoff, and delivery gates remain under construction.
Conclusion
The long-term goal of marketing AI quality engineering is not "zero human involvement" — let AI expand cognitive and execution radius, let engineering mechanisms constrain uncertainty, let humans retain key decision rights, and finally decide releasability by traceable runtime evidence. Only when suggestions can be verified, blocked, corrected, and traced does AI truly become part of the R&D quality system.
Signed-in readers can open the original source through BestHub's protected redirect.
This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactand we will review it promptly.
How this landed with the community
Was this worth your time?
0 Comments
Thoughtful readers leave field notes, pushback, and hard-won operational detail here.
