Beyond the Happy Path: Rigorous Acceptance Testing for Knowledge-Graph Replenishment Pilots
This article details a comprehensive acceptance framework for a knowledge-graph-based replenishment assistant pilot, covering scope definition, version tracking, six distinct verification layers (model, data, suggestions, permissions, execution recovery, handover), independent expected-result calculation, systematic abnormal-case testing, effect measurement including wait and rework time, and capability gating based on test results.
Define Acceptance Scope Before Deciding What to Release
The article opens with a cautionary scene: a project team demonstrates a smooth replenishment flow for Warehouse 1 (query stock, calculate 60 units, obtain approval, create purchase order). When asked about interface timeouts, wrong product mappings, or missing permissions, the team dismisses them as "exceptions to handle later." The author argues that such gaps make the system unsafe for real users.
Before testing, stakeholders must agree on:
Business scope: e.g., only Warehouse 1, bearing 6204, rule-based suggestion, approval by authorized person, creation of purchase order in agreed status in the existing procurement system. Warehouse 2, auto-approval, and goods-tracking are explicitly out of scope.
Pass criteria: which scenarios must succeed, how to handle missing data, timeout thresholds for human handoff, maximum wait for normal tasks. These are written into the acceptance form upfront.
Version baseline: record the ontology version, ID mapping tables, business rules, model and prompt configurations, data sources, interface versions, and the actual accounts used for purchasers and approvers. Using admin accounts instead of real purchaser accounts would miss true permission conditions.
The article references a previous piece on whether ontology projects deliver value, emphasizing that acceptance must be arranged before relying on a single happy-path demo.
Record Each Type of "Pass" Separately
Developers may say "checks passed," but the business owner must ask "which checks?" The article defines six distinct verification layers, each with its own method and limited conclusion:
Definitions & model clarity: Business confirms meaning; tools (e.g., SHACL) validate structure, references, and written constraints. Passing does not prove source data is real or that the program has executed the requirements.
Data retrieval correctness: Trace IDs, units, document states, and data timestamps back to their sources. Passing does not guarantee applicability to another warehouse or dataset.
Suggestion & explanation accuracy: Recalculate with fixed inputs (e.g., 60 units) and verify the reasoning text and quantities. Passing does not mean approval or document creation succeeded.
Creation & exception recovery: Check approval, original request, and purchase order; test timeouts and duplicate requests. Passing does not prove supplier has acknowledged or goods have arrived.
Purchaser handover: Actual assignee locates the document and processes a pending verification task. Passing does not prove long-term time savings.
Run-time improvement: Compare handling time, wait, rework, and maintenance effort. Passing does not attribute all improvement to the ontology alone.
Each item records what was tested, result, confirmer, and evidence location. Untested and failed are distinct; neither should be labeled "to be improved later."
Expected Answers Must Be Independently Determined
Using the article's running example: available stock 120 units, 12 boxes of 10 units each not yet fully received, target 300 units, no other deductions, packaging and MOQ allow 60 units. Expected suggestion = 300 − 120 − 12×10 = 60 units. After valid approval, the created order must match the approval.
Business personnel calculate this expected result from raw data and rules before checking the assistant.
The model may help prepare examples, but the system under test must never provide its own answer as the gold standard; both could make the same mistake.
Distinguish fixed regression samples (re-run against saved data) from live checks (recalculate against current data at that timestamp).
Hold out a set of samples not used during development; vary phrasing, user identity, and input order. Store evaluation answers separately from the knowledge base.
For model-judged critical scenarios, repeat runs per pre-agreed protocol and retain every result, not just the successful one.
Inject Faults to See Where the System Stops
A detailed table lists nine abnormal test cases, each with the fault injected, expected handling, and verification points:
Normal 60-unit request: correct suggestion, valid approval, creation and cross-check against inputs, rules, approval, original request, and order details.
Wrong product mapping: detect at agreed spec-check step, stop suggestion and creation; verify against source product data, mapping table, and check result.
Missing purchase unit: flag missing item, pause quantity calculation, no premature order; check missing-item prompt, call logs, and procurement system.
Approval for 60 units but submission for 60 boxes: block inconsistent submission, verify whether typo or genuine change; check approval content, actual parameters, and interception log.
Unauthorized warehouse query: reject over-privilege query, do not leak data or return zero; verify authorization scope, query call, and response.
Query allowed but no create permission: service-side refusal, not relying on tool-account privileges; check action authorization, tool call, and original system records.
Creation succeeds but receipt lost: recover original request's order, avoid duplicate creation; verify fault setup, request correlation, order count and details.
Query replica not yet updated: show "pending verification," continue polling per agreement, do not claim "not created"; check query timestamp, original request status, and subsequent orders.
Process restart during recovery: resume from saved task record, do not re-issue with new ID; check saved task record, recovery call, and actual result.
Normal cases are equally important: a fully valid request that the system refuses is also a failure. Proving "no creation" requires checking the source system's processing and query conventions, not just the assistant's logs. Unclear outcomes are recorded as "not yet confirmed," not pre-filled as passed.
Each test records version, inputs, expected, actual, and evidence location. Simulated interfaces are marked as such; they verify calling logic but not the real procurement system's behavior. Faults are first run in an isolated, approved test environment; later real-system trials use restricted accounts, items, quotas, time windows, and designated exception handlers. Both phases are documented separately.
Measure Pilot Effect Including Wait and Rework
Track total elapsed time from purchaser's question to correct order in hand, broken down into manual handling, approval wait, verification, and rework. A suggestion that returns in seconds but stalls in approval for days has not sped up the process.
Define upfront which tasks are in scope and the observation window.
List failed, human-escalated, and incomplete tasks at window end — do not cherry-pick only successful orders for average time.
Keep product scope, quality requirements, and personnel comparable to the old process.
Log developer time spent back-filling data, investigating exceptions, and modifying rules. Concurrent interface or policy changes mean the improvement is systemic, not solely from the ontology.
Correct order creation is the pilot's output; reduced stockouts depend on later delivery, demand, and supply factors and cannot be inferred from creation rate alone.
The article cites NIST AI RMF's Measure function: document test sets, metrics, evaluation tools, assess under near-deployment conditions, and monitor operational performance. The acceptance report must explain which samples were used, how they differed from production, and what emerged during trial use.
Gate Capabilities by Failed Checks
If read-only queries and suggestions pass but post-creation exception recovery is unverified, enable query and suggestion but disable creation entry points and corresponding tool calls — enforced server-side, not just UI hiding.
If queries themselves leak data, restrict the relevant queries.
If product mappings are wrong, pause dependent suggestions and creation.
Restrict exactly where the failure reaches; avoid blanket "read-only for now."
Each failed item documents impact, owner, and retest date. After fix, re-run the failed samples and regression-check related happy paths before formal re-approval. If a stakeholder accepts a risk temporarily, the failure record remains; a signature does not equal a pass.
Post-release, monitor data latency, backlog of pending verifications, and whether exceptions are handled on time. Any change to interfaces, rules, mappings, or model configs triggers impact analysis on prior conclusions and supplementary testing before continued use.
Summary
At the acceptance meeting, the team should produce records showing: which order was found after a timeout, at which step a wrong mapping was caught, why an unauthorized user could not create an order. Untested items are explicitly noted with the capabilities they block.
The acceptance conclusion must tell the procurement lead who can use the assistant, for which tasks, and who owns exceptions. A single smooth demo cannot answer these questions.
Trial operation will generate alerts and correction logs. The next article will examine which of those are reusable for future tasks and which were one-off fixes.
References
W3C, Shapes Constraint Language (SHACL), Section 3 Validation and Graphs. Used to illustrate data validation under given constraints and reporting; does not prove business facts or authorized actions. https://www.w3.org/TR/shacl/
NIST, AI RMF 1.0 Core, Measure. Used to illustrate evaluation records, near-deployment testing, and operational monitoring; the acceptance matrix in this article is author-designed. https://airc.nist.gov/airmf-resources/airmf/5-sec-core/
Signed-in readers can open the original source through BestHub's protected redirect.
This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactand we will review it promptly.
How this landed with the community
Was this worth your time?
0 Comments
Thoughtful readers leave field notes, pushback, and hard-won operational detail here.
