Agent Citing Stale Experience? How to Audit, Retire, and Replace Obsolete Knowledge Cards

Introduction: The Problem of Outdated Experience Cards

On a production line that has not changed, an AI agent repeatedly suggests troubleshooting steps that field personnel override. The agent cites a familiar experience card linking dimensional deviation with a specific sound pattern, but the tool supplier has since been replaced. The article uses this hypothetical extension of a quality anomaly series to explore how to handle knowledge that may no longer apply.

First Confirm Where "Inaccuracy" Occurs

Field complaints that "the agent is recently inaccurate" are an investigation entry point, not a root cause. Teams must examine the specific task: which records were used, which experience version was retrieved, what suggestion was made, why the field overrode it, and what subsequent inspection confirmed. A rejected suggestion does not automatically mean the suggestion was wrong; an inconclusive inspection does not prove a false alarm.

For the technical team, this is a RAG (Retrieval-Augmented Generation) knowledge maintenance issue and part of LLMOps (evaluation, release, and operational maintenance of LLM applications). The field needs clear authority over who confirms an experience card's applicability. Automated ingestion can move materials but cannot replace verification; unconfirmed guesses that enter the retrieval scope may be repeatedly cited even if the interface is healthy.

When comparing before-and-after performance, verify product mix, operating conditions, task volume, and observation window. A recent increase in complex tasks may lower overall pass rates; looking only at closed issues may miss unresolved difficult cases. Group by comparable conditions, retain unconfirmed results, and do not equate a string of complaints with an accuracy metric.

Check for missing records, unit changes, and correct equipment-batch associations. Evidently's developer technical note places data quality checks before drift handling: collection or code faults can also cause distribution shifts, and fixing inputs does not necessarily require model retraining. [1]

At least three distinct changes must be distinguished:

Data Drift : Input distribution changes (e.g., new product mix appears more often), but the original judgment may remain valid in each original condition.

Concept Drift : The relationship between inputs and actual outcomes changes under the same task and result definition. This is the core problem discussed in related research. [2]

Business Rule Change : New inspection requirements alter applicable tolerances; first confirm the official version and its effective scope.

These changes can occur simultaneously. A drift alert only triggers investigation; without reliable results, one cannot conclude diagnostic rules have failed solely because inputs changed. Also check whether the model, prompts, or retrieval index were recently upgraded, to avoid blaming field knowledge for system changes.

When Should This Experience Be Reviewed, and Who Decides to Deprecate?

Previous articles discussed experience collection. Once in operation, it is insufficient to keep only "applicable equipment"; one must know what changes trigger a review and who handles it. For each experience card, at minimum the following engineering management fields should be populated (they are not a mandatory template from any standard):

Applicable Scope : Which equipment, products, tools, and conditions are covered; which combinations remain unverified.

Basis and Version : Source verified cases, referenced process or manual version, effective date.

Review Triggers : How tool or process changes, reliable counterexamples, repeated overrides, or source document replacements generate notifications.

Current Usage : Whether the card may prompt investigation, participate in an approved judgment, or is prohibited for certain uses.

Responsibility and Deadline : Which professional role confirms validity, who leads evidence gathering, when to revisit pending items.

Impact and Alternatives : Which tasks have referenced it, what fallback path applies when restricted, where the replacement version resides.

When process change orders can link to experience cards, affected knowledge should be proactively checked during change review, not waiting for the agent to err. If automatic linking is not yet feasible, the change owner can maintain an affected-items list.

Reviews can be event-triggered or periodic based on risk. Long-stable manual entries and frequently changing equipment experiences need not share a single "shelf life." Every pending review item must have an owner, a review deadline, and a current usage decision; if the deadline passes without resolution, escalate, restrict, or suspend per pre-agreed risk rules — do not assume an extension permits continued use.

Equipment running or under maintenance status should be updated from operational records; business concept definitions, case-supported judgments, and permitted process actions are each maintained by their respective responsible parties. Do not modify ontology definitions because equipment status changed, and do not substitute periodic knowledge-base refreshes for real-time status verification.

Deprecating Old Experience Does Not Mean Deleting History

If a new tool combination is unverified, a reasonable approach is to restrict the card's use for that combination rather than deleting past cases.

Track review progress and usage status separately. Review progress: "pending verification", "under verification", "completed". Usage status: "allowed", "scope-limited", "suspended", "replaced". Separation clarifies "under review but still allowed for old conditions" instead of letting "pending review" become a vague pass.

With reliable risk evidence, an authorized role may suspend relevant uses per established procedure before root cause is fully known. The suspension applies to the knowledge or assistive capability, not to equipment shutdown decisions made unilaterally by knowledge maintainers.

Deprecation must propagate to where knowledge is actually used. If the original is marked "invalid" but retrieval indexes, caches, and ongoing conversations still contain old fragments, the agent may continue citing them. The system should retain knowledge identifiers and versions, check current allowed scope at retrieval and generation entry points, and evict or rebuild contexts containing deprecated fragments — not rely on the model reading "please ignore old content." For actions, existing execution qualification checks must still apply.

Historical records should be retained per permission and retention requirements, documenting what a judgment referenced at that time. W3C PROV-O provides provenance expressions such as revision relationships and invalidation timestamps, usable for describing version links. [3] Entity invalidation in PROV-O does not directly equal "restrict only a certain condition"; partial restrictions must be recorded separately. These expressions themselves do not evict caches or block calls; enforcement still depends on application mechanisms. Small-scale scenarios can use version tables and reference logs without building a graph database.

Human Override Should Not Immediately Overwrite the Knowledge Base

When field personnel say "after the tool change this experience card cannot be used," first create a traceable feedback record: associated task, changed judgment, basis, and which facts were unclear at the time.

Specialists then verify tool, machining period, inspection results, and the original card's applicability conditions. If the case merely exceeds the verified scope, record "applicability unknown"; only with sufficient counterexamples should applicability conditions be adjusted or the judgment withdrawn. Do not label both situations as "old experience wrong."

Next decide what to fix: wrong batch mapping → fix mapping; new manual published but not retrieved → fix source and index; process requirements changed → business owner confirms rule and effective scope; evidence complete but model still misjudges → evaluate prompts, thresholds, or model choice. When multiple causes coexist, assign a lead for each fix target.

A single change ticket can chain the process: feedback logging, fact verification, cause localization, asset modification, independent validation, approval and release. At minimum attach affected versions and tasks, temporary usage restrictions, candidate revisions, validation materials, approver, and release time, avoiding each team handling only a chat message.

The agent can organize materials, surface similar feedback, and draft revision diffs, but candidate text must not automatically overwrite approved knowledge. When experts disagree, retain alternative interpretations and pending evidence; do not use like counts or majority overrides as final fact.

Unverified materials can be placed in a logically isolated staging area, excluded from production judgment retrieval. The diagram below illustrates the change flow; it does not require a separate database nor imply every feedback must modify the model.

Also prevent feedback from coming only from "system alarmed" tasks. Appropriate sampling of non-alarming tasks reveals omissions; keep uncompleted inspections as unconfirmed. Otherwise the system may become better at reducing complaints without truly improving missed detections.

New Version Must Prove Improvement and Show No Regression

Problem-discovered cases suit reproduction and revision but cannot serve as the sole acceptance set. Designated validators retain unadjusted materials to separately check new conditions, still-supported old conditions, and cases with missing records, source conflicts, or indeterminate outcomes.

For case retrieval, verify that applicable materials are found and that deprecated content is blocked from serving as current judgment basis; historical audit may still view old versions within permitted scope but must label purpose and status. For diagnostic assistance, check candidate judgments and follow-up requirements; for action requests, verify continued adherence to permissions and state conditions. Do not substitute "referenced new version" for business outcome validation.

If quality requirements have formally changed, old cases cannot mechanically reuse old answers: retrospective decisions are evaluated against rules effective at that time; current use cases are evaluated against rules effective for that object and batch — not simply the latest published file. When historical materials lack fields required by new rules, supplement evidence or mark as unevaluable. Independent validation sets must be maintained under task definition control, not temporarily altered to let a new version pass.

At release, record the actual dependent versions of experience, rules, index, prompts, and model, and verify the version combination consumers actually use; if combinations are incompatible, do not allow related uses — changing only the knowledge file does not constitute a completed switch. First observe in historical replay or shadow run, then trial within approved scope; shadow paths must not write production work orders, states, or control requests. Compare misjudgments, missed detections, handovers to human, and review burden — not just overall correctness.

When new samples are insufficient, explicitly state evidence gaps, restrict usage, and set a supplementation deadline. Do not assume a new version is more reliable simply because the old version had issues.

Don't Forget In-Flight Tasks During Version Switch

Suppose a morning analysis referenced the old card, the old card is suspended in the afternoon, and the task still awaits confirmation. Only letting new tasks read the new version leaves this work order unhandled.

Placing this on a timeline clarifies the sequence (times below illustrate order only).

Using reference logs, identify affected tasks and have the responsible person decide which need re-evaluation, evidence supplementation, or notification. Unexecuted suggestions must re-check qualifications and basis; sent requests with unknown outcomes must first verify execution status; already executed field actions enter existing review and disposal procedures — physical consequences cannot be assumed revoked because knowledge was rolled back.

Routine revisions can handle in-flight tasks per approved compatibility strategy; emergency deprecation must be able to stop deprecated basis from driving further actions. Tasks retain their original referenced version while adding a re-evaluation record, not silently rewriting historical answers to the latest version.

Rollback is not always to the "previous version." If the old version violates current process requirements or is proven inapplicable, it cannot be restored. When no qualified rollback target exists, narrow the assistive scope or fall back to a verified human path; human handover capacity is confirmed per the previous article's method.

Business knowledge owners confirm applicability; data and model personnel fix their respective assets; validators check evidence; system owners enforce cutover and deprecation. Before closing the change, verify that deprecation requirements are effective, affected tasks are resolved, and monitor for recurrence during an agreed observation window. Low-frequency anomalies not recurring cannot alone prove fix effectiveness; with insufficient samples, retain validation conclusion limitations and assign follow-up re-evaluation responsibility.

Conclusion

Factories need not only an ever-growing knowledge base but also a way to know which experiences are still usable today. Tracing a single override back to facts, responsibilities, versions, and unfinished tasks gives old experience a chance to be correctly retained, restricted, or replaced.

When knowledge expires, the interface may still return success. The technical team must make deprecation and cutover actually take effect; the business owner must specify what is stopped, why, and which scenarios remain usable.

The next article shifts focus to the field: how does a person wearing gloves, in a noisy environment, executing a task, actually receive this information and complete the feedback loop?

References

[1] Evidently: My data drifted. What's next? How to handle ML model drift in production. Developer technical note, covering data quality investigation, drift investigation, and handling options; not proof of manufacturing field effectiveness. https://www.evidentlyai.com/blog/ml-monitoring-data-drift-how-to-handle [2] Gama, J., et al. (2014). A survey on concept drift adaptation. ACM Computing Surveys, 46(4), Article 44. Verified publisher abstract only, used for concept explanation of input-target relationship change; no claim of full algorithm and experiment review. https://doi.org/10.1145/2523813 [3] W3C (2013): PROV-O: The PROV Ontology. Cites provenance expressions such as wasRevisionOf and invalidatedAtTime; does not treat ontology relations as execution mechanisms for knowledge deprecation or cache invalidation. https://www.w3.org/TR/prov-o/