Terminology Normalization Beyond Similarity: Coverage, Routing, Drift & Regression Testing

The previous article discussed the relationship between stable business schemas, reference ontology catalogs, and dynamic normalization, and provided an auditable mapping result contract.

However, returning a standard URI for a raw term only proves the pipeline runs; it does not prove the mapping is correct, nor that it will remain stable over time.

A production-grade normalization system must answer: can the correct concept enter the candidate set? Which results can be auto-confirmed? Which require manual review? How to detect drift when catalogs or inputs change? How to prove existing capabilities have not regressed after algorithm upgrades?

Terminology normalization is not a single similarity query, but a continuous governance mechanism built around candidate recall, risk routing, human confirmation, drift attribution, and regression testing.

Coverage Rate Is a Routing Signal, Not a Quality Score

The article emphasizes the role of coverage rate: if input categories find candidates above a recall threshold in the reference catalog, the catalog has some coverage for that batch; a drop in coverage may signal new drugs, new abbreviations, or input distribution shifts.

This metric has value but is easily misused.

Suppose 1,000 raw terms yield candidates above the recall threshold for 900 terms — a 90% candidate coverage rate. That does not mean 900 mappings are correct. A broad but poorly bounded catalog may achieve high candidate coverage while producing many false matches.

Evaluation must separate at least the following metrics:

Candidate Coverage Rate — How many inputs find candidates meeting the recall threshold. Cannot prove candidates are correct.

Top-K Recall Rate — Whether the correct concept appears in the top K candidates. Cannot prove the top-ranked candidate can be auto-accepted.

Auto-Mapping Coverage Rate — How many inputs the system auto-confirms. Cannot prove auto results are sufficiently accurate.

Auto-Mapping Accuracy — How many auto-confirmed results are correct. Cannot prove the uncovered long tail is handled.

Manual Review Rate — How many inputs enter the human queue. Cannot prove review is effective or cost-acceptable.

Unresolved Rate — How many inputs fail to produce a valid mapping. Cannot distinguish whether the cause is new concepts, data quality, or catalog gaps.

Projects must not chase higher automation rates alone. For failure types affecting equipment downtime, production scheduling, or safety handling, it is better to keep "unknown" and route to human review than to create a false sense of certainty with a high-similarity candidate.

Review Routing Cannot Rely on a Single Score Threshold

More reliable routing should consider matching evidence, candidate gaps, term type, and business risk simultaneously.

Auto-Confirm — Authoritative code or approved alias exact hit; type, version, and scope consistent. Action: Write to confirmed mappings, retain sampling review.

Controlled Auto-Confirm — Multiple evidences align, first candidate leads clearly, low-risk category. Action: Confirm per scenario thresholds, continuously monitor error rate.

Manual Review — Short abbreviations, multiple close candidates, insufficient context, or high-risk concepts. Action: Show original text, context, candidate definitions, and differences.

Keep Unmapped — No valid candidate, catalog version inapplicable, or insufficient input evidence. Action: Retain original value, route to terminology backlog or data repair.

Model Change Candidate — Large volume of valid inputs consistently cannot be carried by existing business concepts. Action: Gather business evidence, initiate separate model change process.

The manual review interface must not merely show "Candidate A, score 0.91". Reviewers need to see original context, candidate definitions, source catalog, similar candidates, historical mappings, and potentially affected business rules. Otherwise, machine uncertainty is simply shifted to humans.

When Drift Occurs, First Determine What Changed

Declining candidate coverage, rising human rejections, or increasing online errors are only runtime signals; they do not directly imply "the ontology needs updating."

At least four drift categories must be distinguished:

Input Terminology Drift. New supplier codes, new abbreviations, or new textual expressions appear. Usually addressed by adding aliases, catalog entries, or mapping rules.

Reference Catalog Drift. Standard concepts are added, deprecated, merged, or split. Requires catalog version upgrade and re-running affected mappings.

Mapping Strategy Drift. Model, vector representation, thresholds, or re-ranking logic change, causing the same input to yield different candidates. Requires independent regression validation.

Business Semantic Drift. Real-world object boundaries, states, or rules genuinely change, and the existing business schema can no longer carry them. Only then should model evolution be triggered.

These four changes require fixes at different layers. Treating a terminology gap as a business concept upgrade causes schema bloat; treating a real business change as a mere alias masks the fact that states and rules have shifted.

Before every catalog or strategy upgrade, impact scope must be identified: which confirmed mappings will change, which data needs re-normalization, which queries, rules, and agent contexts depend on those results. New candidates must not silently overwrite historical mappings; historical judgments must retain the catalog and strategy versions used at the time.

How Should Terminology Normalization Be Accepted

A demo that returns URIs does not mean the system is production-ready. At minimum, a set of real samples confirmed by business owners is needed, covering normal expressions, abbreviations, misspellings, contextual ambiguities, supplier variations, new terms, and cases with no correct candidate.

Acceptance checklist can include:

Whether business schema clearly defines which objects and fields allow normalization.

Which reference catalogs, concept types, and versions each term category may query.

Whether Top-K candidate recall covers confirmed correct concepts.

Whether auto-confirm results meet agreed accuracy per business risk level.

Whether short terms, abbreviations, and similar concepts have dedicated challenge sets.

Whether the system stably rejects mapping when no correct candidate exists, instead of forcing a result.

Whether original value, candidates, method, score, version, and confirmation record are fully traceable.

Whether golden samples can run regression tests after catalog or algorithm upgrades.

Whether online rejections, unresolved cases, and sampling review results feed into attribution and improvement queues.

Whether downstream queries and agents can distinguish original facts, candidate mappings, and confirmed mappings.

Test sets must also be isolated by data source and time. Randomly sampling near-duplicate expressions from the same supplier and vocabulary easily overestimates system capability against new equipment, new versions, and new terminology.

Not All Projects Need Dynamic Terminology Normalization

This analysis targets business applications that must carry concrete objects, processes, rules, and actions. If the system itself is a reference ontology editor, publisher, exchange, or general reasoning platform, or if business objects are highly isomorphic with standard concepts and downstream can accept the ontology's complexity and version cadence, the reference ontology may directly serve as primary storage or query schema. Even then, application-specific constraints, permissions, and service contracts are usually needed.

The question is not "can the standard ontology be used directly," but whether it truly aligns with the current application's object boundaries, query patterns, and evolution responsibilities. The article opposes importing large reference ontologies wholesale into ordinary business applications without this judgment.

If a scenario has a single source system, stable code range, and strict database foreign-key constraints, maintaining an approved mapping table is often more reliable than building a vector catalog and online normalization service.

If input volume is small, business risk is high, and all new terms already require expert approval, manual registration with dual review can be prioritized without pursuing automatic matching.

Scenarios truly warranting dynamic normalization capability typically exhibit these characteristics:

Multiple systems, suppliers, or institutions use different codes and natural-language descriptions for the same concepts.

Terminology and reference standards update continuously; static mapping maintenance has become a delivery bottleneck.

Downstream queries, analytics, and agents rely on unified concepts for cross-source correlation.

Original expressions must be retained, and mapping results must be used with confidence boundaries and versions.

The team can build golden samples, manual review, regression testing, and drift monitoring.

If the last item is infeasible, so-called "dynamic" normalization often only adds invisible semantic errors.

Build a Minimum Closed Loop Starting from One Terminology Type

Enterprises need not import all industry ontologies at once, nor normalize all business fields in one go. A more feasible path:

Select one terminology type with clear value and verifiable errors, e.g., equipment fault codes.

Freeze the current business schema; define target concept types, use cases, and non-confusable boundaries.

Choose an authoritative, scope-controlled reference catalog; record source, license, version, and update mechanism.

Establish exact codes and approved aliases first, then gradually add lexical, vector, and contextual candidate recall.

Calibrate routing rules with real samples; define conditions for auto-confirm, manual review, and keep-unknown.

Publish versioned mapping results, letting downstream consume only states matching their trust policies.

Monitor coverage, accuracy, review rate, and unresolved causes; route issues to data, catalog, mapping strategy, or model change processes.

The goal of this path is not to make the system find an answer for every word, but to make every answer explain: what was the original expression, why map here, based on which version, who confirmed, and why it did not auto-pass when uncertain.

Summary

Standard ontologies and business schemas do not conflict; they simply bear different responsibilities.

Business schemas provide stable contracts for applications, queries, rules, and agents; reference ontologies provide standard identifiers, definitions, and reusable terminology; dynamic normalization sits between them, absorbing new expressions, retaining candidate evidence, and managing uncertainty.

What truly needs dynamic updating are traceable mapping records and review queues. Only when business reality itself changes — when existing objects, states, relations, or rules can no longer express it — should business model evolution be initiated.

Interoperability is not about moving the standard ontology wholesale into the application, but about enabling expressions from different sources to be controllably connected to shared concepts without breaking business contracts.

The next article will discuss: when multiple agents cannot share complete interfaces and global semantics at design time, what can runtime negotiation solve, and which deterministic engineering contracts cannot be bypassed.

Agents can dynamically negotiate semantics, but cannot bypass engineering contracts: what Liquid Interface actually changes.