Evaluating Ontology Intelligence Projects: 6 Dimensions Beyond Concept Counts

The previous article discussed how ontologies evolve through versioning, change management, impact analysis, and runtime feedback — a practice called KnowledgeOps. Once an ontology model is released, accepted, and operational, the final question remains: does this ontology intelligence system create real business value and justify continued investment?

Many projects showcase concept counts, relationship counts, triple scale, data ingestion volume, and API counts. These metrics reflect construction effort and asset size but cannot prove business users actually rely on the system, that agent judgments are more accurate, actions more reliable, or risks more controllable.

The evaluation target of an ontology intelligence project is not a single ontology graph or knowledge base, but the end-to-end business result formed by semantics, data, decisions, actions, and governance within a concrete business scenario.

Why Concept Counts and Triple Scale Cannot Represent Success

Concept, relationship, and triple counts have value for observing model scale, data loading, and construction progress, and can help detect abnormal growth or duplicate modeling. However, they are asset-inventory metrics, not project-success metrics.

An ontology may contain tens of thousands of concepts yet fail to cover the questions the business actually needs answered. A knowledge graph may hold billions of triples but, due to object-identity errors, stale relationships, and missing evidence, cannot support reliable decisions.

Conversely, a small model built only around customers, receivables, contracts, disputes, and disposal actions — if it stably supports a credit-warning closed loop — may be more valuable than a massive enterprise ontology nobody uses.

Similarly, query counts and agent invocation volumes only indicate usage activity. High-frequency invocation of wrong results does not create business value; extensive manual re-checks and retries may even increase cost and risk as volume grows.

Therefore, concept counts, triple counts, API counts, and invocation volumes can be retained as diagnostic and operational indicators but must not bear the burden of project success/failure conclusions.

Model Acceptance and Project Evaluation Are Not the Same

Model acceptance verifies whether a specific release unit and its target runtime implementation satisfy agreed behaviors under specified data, rules, and test environments. Project evaluation goes further: it observes whether these capabilities, once in real work, continuously change business outcomes and whether cost and risk are acceptable.

Comparison

Primary Object: Model acceptance targets the model release unit and its runtime implementation; project evaluation targets the complete ontology intelligence capability in a business scenario.

Primary Evidence: Model acceptance uses capability questions, test data, rules, contracts, acceptance cases; project evaluation uses real traffic, business outcomes, runtime events, cost, user behavior.

Time Range: Model acceptance covers a specified version and acceptance window; project evaluation spans continuous operation cycles across multiple version changes.

Main Conclusion: Model acceptance decides whether the current version is usable within declared scope; project evaluation decides whether the project should expand, optimize, restrict, or stop.

Passing model acceptance is a prerequisite for controlled production evaluation, but not proof of project success. A model may pass all tests yet business users refuse to adopt it, manual review costs remain high, or no business metric improves — the project can still fail.

Conversely, a project cannot bypass model acceptance by claiming "good business effects." Semantic errors, untraceable evidence, or unauthorized high-risk actions cannot be accepted just because of short-term gains.

Define Evaluation Unit, Business Baseline, and Attribution Method First

An ontology intelligence project cannot compute a single global score detached from scenarios. Before evaluation, clarify:

Which role performs which business task;

Where the process starts and what business result it ends with;

Whether ontology intelligence participates in query, judgment, recommendation, or action execution;

What the original working method, duration, quality, risk, and cost baseline are;

Target values, measurement cycles, scope, and owners;

Which ontology models, foundation models, data, rules, prompts, action contracts, and agent versions this evaluation binds to.

The same platform may support multiple scenarios, but contract review, equipment warning, and credit disposal have completely different value indicators, risk levels, and cost structures. Evaluate each scenario separately, then aggregate to the portfolio level — do not mask failing scenarios with an average score.

Without a business baseline, it is hard to prove "the project brought improvement." If post-launch processing time is 20 minutes, only comparison with original processing time, task complexity, and manual effort can determine progress or regression.

Effect attribution is equally critical. Business results may be influenced simultaneously by ontology model, data quality, rule adjustments, foundation model upgrades, process changes, and personnel training. Where possible, use phased rollout, control groups, before/after comparisons of similar tasks, or version comparisons; when variables cannot be strictly isolated, state assumptions and uncertainties explicitly — do not attribute all improvement to the ontology.

To further judge the ontology's incremental contribution, compare a baseline solution without a unified semantic layer against the ontology-semantic solution under identical task, data, and model configuration, or use component ablation to measure effect changes after removing object relations, rule semantics, or evidence constraints. However, component contribution cannot replace end-to-end project results; local metric improvements must still be confirmed in real business processes.

Set Safety Thresholds First, Then Six-Dimension Evaluation

Ontology intelligence projects should not compute a weighted total score first and then decide pass/fail. High-risk issues must form non-compensable entry thresholds.

Typical thresholds include:

Unauthorized data access and sensitive information leakage;

Definite conclusions given without critical evidence;

Execution of high-risk business actions without approval;

Repeated execution causing payment, limit adjustment, notification, or other business side effects;

Inability to audit key decisions, manual modifications, and action results;

Production results that cannot be linked to the ontology model, foundation model, data, rules, prompts, and action contract versions used at the time.

Specific threshold values depend on scenario risk. For low-risk knowledge queries, a certain proportion of answer failures with human fallback may be allowed; for high-risk actions like payments, limit adjustments, and account restrictions, unauthorized execution typically requires zero tolerance.

Only after mandatory thresholds pass does the project enter comprehensive evaluation across six dimensions: business value, semantic quality, decision reliability, action reliability, security governance, and operational cost.

The six dimensions form a complete inspection framework; not every scenario must contain all capabilities. Projects providing only query and explanation may mark action reliability as "not applicable" and state boundaries; but if the project claims to recommend or execute actions, it cannot use "not applicable" to avoid corresponding evaluation.

Dimension 1: Has Real Business Value Occurred?

Business value must return to the project's original value narrative, checking whether work methods and business results have changed.

Focus on:

Whether single-task processing time and manual effort decreased;

Whether risk detection is earlier, missed reports, rework, and waiting reduced;

Whether business personnel continuously use the system in real tasks, not just during demos;

Whether business personnel's adoption and review of conclusions improved, e.g., shifting from line-by-line re-check to risk-stratified review;

Whether recommendations actually enter subsequent disposal, not just sit on an unseen page;

Whether verifiable gains, avoided losses, or risk exposure improvements actually occurred.

Business metrics must stay as close to the process endpoint as possible. "Number of alerts generated" is not final value; whether alerts are confirmed, drive timely disposal, and reduce losses is closer to business outcome.

User trust cannot be measured by satisfaction surveys alone, nor can "reduced manual review" itself be treated as success. Combine adoption rate, override rate, spot-check results, and complaints to distinguish reasonable trust built on stable performance and traceable evidence from blind reliance on automation; high-risk tasks must retain agreed human confirmation points.

For hard-to-quantify value like avoided losses or early risk detection, use conservative intervals and clear assumptions — do not package unprovable potential gains into a precise ROI.

Dimension 2: Does Semantic Quality Support Real Operations?

This dimension no longer counts how many concepts were built, but observes effective coverage and correctness of business semantics in real traffic.

Key points include:

Coverage of high-priority capability questions in real tasks;

Correctness of object identity merging, attribute and relationship mapping;

Proportion of key conclusions traceable to evidence and sources;

Occurrence rate of unmapped objects, unknown states, and evidence conflicts;

Frequency of business personnel corrections to concepts, states, and relationships;

Degree to which the same semantic asset is reliably reused across different services, agents, or scenarios.

Many unknown states do not necessarily indicate a poor model; they may reflect the system honestly exposing data gaps. The real danger is when the model cannot express "unknown" or the runtime system stably interprets "no data" as "business fact does not exist."

Semantic reuse cannot be measured by reference count alone. Only when reuse reduces duplicate modeling, terminology conflicts, and integration costs does it constitute verifiable value.

Dimension 3: Are Decision Effects Reliable?

For query-and-explanation-only scenarios, this dimension evaluates answer, evidence, and controlled refusal quality; when the system further participates in risk judgment, classification, or disposal recommendations, metrics must match the corresponding decision consequences.

Decision effect cannot rely on a single average accuracy. Different errors have different business consequences — missing a high-risk customer is not equivalent to a false alarm on a regular customer.

Combine scenario evaluation with:

Accuracy, recall, false positive rate, and false negative rate split by risk level;

Whether the proportion of active refusal or human handoff when evidence is insufficient is reasonable;

Business expert override rate, and whether overrides stem from semantics, data, rules, or model;

Whether structured conclusions and evidence are stable across repeated runs of similar tasks;

Whether cited facts, rules, and documents genuinely support the judgment;

Whether degradation occurs after model, prompt, data, or rule changes.

If the decision chain includes a large model, conduct repeated evaluation on fixed test sets and real sampled tasks, focusing on structured conclusions, evidence citations, and tool selection — not requiring identical wording each time.

Projects must also examine "controlled failure" quality. A system that knows when it cannot judge and hands the task to the right person is often more reliable than one that forces answers in all scenarios.

Dimension 4: Are Actions Truly Completed and Not Out of Control?

An agent API returning 200 does not mean the business action succeeded. Action evaluation must trace to the business system's formal result or entry into explicit failure, compensation, and human takeover states.

Evaluate:

Among tasks meeting execution conditions, how many correctly formed candidate actions;

Whether actions requiring approval all went through correct role confirmation;

Whether action parameters, object versions, and preconditions were correct;

Whether successful execution obtained business receipt and updated subsequent state;

Whether duplicate requests were intercepted by idempotency mechanisms;

Whether timeouts, partial failures, and external system anomalies can recover, compensate, or hand off to humans;

Whether action failures reversely pollute business object state and subsequent decisions.

The denominator of action success rate must be clear. A more reasonable definition: among actions that meet execution conditions and actually enter execution, the proportion that complete business results and obtain valid receipts — not using API call count as denominator.

Dimension 5: Is Security and Governance Continuously Effective?

Safety thresholds address "can it continue running"; this dimension further observes whether governance capabilities remain effective in real use.

Monitor:

Whether data and tool access always comply with role and scenario permissions;

Whether privilege escalation, prompt injection, and anomalous tool requests are detected and blocked;

Whether key conclusions, versions, evidence, approvals, and action logs are complete;

Whether human confirmation points are bypassed and manual modifications are traceable;

How security incidents, near-misses, and user complaints are discovered, classified, and closed;

Whether governance boundaries are synchronously updated after model, rule, permission, or interface changes.

"No accidents during operation" alone cannot prove system safety. Must combine actual attack or privilege escalation attempts, red-team test coverage, interception results, and audit sampling to judge whether security mechanisms truly work.

Security metrics should retain event count, severity, and exposure scope. A single high-risk privilege escalation cannot be diluted by large volumes of low-risk successful queries.

Dimension 6: Are Operational Costs and Unit Economics Sustainable?

Ontology intelligence project costs go beyond LLM token fees and include:

Data ingestion, mapping, quality repair, and object projection maintenance;

Model inference, retrieval, tool invocation, and infrastructure resources;

Manual review, exception takeover, and business expert maintenance;

Version governance of models, rules, action contracts, and test sets;

Failure recovery, security audit, migration, and compliance costs.

More meaningful than "cost per model call" is total cost per controllably completed business task . Only when the task truly completes, evidence is traceable, and risk stays within agreed boundaries does this cost become comparable.

Unit economics must compare verifiable value per task against this total cost. When value cannot be directly monetized, at least set a business-acceptable cost ceiling and explain estimation assumptions — do not declare economic viability just because token unit price drops.

Operational quality should also include availability, response time, failure recovery time, change delivery cycle, and regression testing cost. If every policy adjustment requires months of manual changes, or the system cannot be maintained without the core project team, scaling is difficult.

The economic value of ontology reuse should manifest as declining marginal modeling, data integration, and governance costs for new scenarios — not merely claiming "the model is reusable."

Don't Let a Single Composite Score Mask Real Problems

The six dimensions can be summarized in a scorecard or radar chart, but must not collapse into a single final score.

A complete scorecard should at least record:

Scenario & Metric: Which role, process, and business goal the metric serves

Metric Definition: Numerator, denominator, statistical caliber, exclusion conditions

Baseline, Target, Actual: Pre-launch baseline, agreed target, current result

Measurement Window: Time range, sample size, business volume

Version Context: Which ontology models, foundation models, data, rules, prompts, action contracts, agent versions used

Evidence & Owner: Source of result, who confirms and interprets

Metric Type: Mandatory threshold, phase gate, or continuous observation metric

Safety, compliance, critical evidence, and high-risk actions are mandatory thresholds that cannot be offset by high scores in other dimensions. Business value, efficiency, accuracy, and cost metrics can be weighted per scenario to assist version comparison and trend observation, but raw metrics, anomaly distributions, and mandatory thresholds must be preserved simultaneously.

Beyond averages, retain distributions across different customers, task types, risk levels, and anomaly scenarios. High average accuracy does not guarantee absence of systematic failures in a few high-risk tasks.

Build a Project Scorecard Using the Credit Warning Scenario

Continuing the "identify overdue risk and form credit disposal recommendation" scenario, project evaluation can be organized as follows:

The last column distinguishes two constraint types: mandatory thresholds trigger immediate restriction or stop of relevant capabilities; phase gates not passed prohibit entering next phase or expanding scope. Neither can be offset by composite scores, but handling differs.

Scorecard

Business Value — Core question: Whether risk is detected earlier and drives disposal. Example metrics: Review duration, alert lead time, manual effort, actual disposal rate. Phase gate: no value signal over multiple evaluation cycles → no scaling.

Semantic Quality — Core question: Whether customers, receivables, contracts, disputes are correctly linked. Example metrics: Capability question coverage, identity merge accuracy, evidence traceability, unknown state rate. Mandatory threshold: critical object identity error, or high-risk conclusion untraceable to key evidence.

Decision Effect — Core question: Whether alerts and disposal recommendations are reliable. Example metrics: Stratified recall, false positive, false negative, human override rate, controlled refusal rate. Mandatory threshold: high-risk false negative rate exceeds agreed tolerance.

Action Reliability — Core question: Whether recommendations are confirmed and produce correct business results. Example metrics: Approval coverage, business completion rate, duplicate side effects, failure recovery rate. Mandatory threshold: unauthorized execution of high-risk action, or uncontrolled duplicate side effects.

Security Governance — Core question: Whether data, judgments, actions stay within authorized boundaries. Example metrics: Privilege escalation interception, audit coverage, critical evidence missing, high-risk events. Mandatory threshold: privilege escalation, sensitive leakage, or critical audit break.

Operational Cost — Core question: Whether the capability can run and scale long-term. Example metrics: Total cost per controlled task, manual review duration, availability, change delivery cycle. Phase gate: unit task total cost continuously exceeds agreed ceiling → no expansion.

Suppose alert recall improves significantly but many results require manual re-investigation due to insufficient evidence, causing per-task total cost to rise — do not simply announce "model effect improved." The team must diagnose whether the issue stems from data coverage, semantic mapping, rule boundaries, or business process, and re-evaluate after remediation.

If business value appears but unauthorized auto limit adjustment occurs, even if all other dimensions meet targets, the project must immediately restrict related actions — not continue expanding with a high composite score.

This scorecard's purpose is not to prove all metrics higher is better. Too low a controlled refusal rate may mean the system forces answers; too high may indicate insufficient data and model coverage; it must be interpreted together with misjudgment, human takeover, and business risk.

Different Phases Require Different Decisions

Many AI projects fall into "Pilot Purgatory": impressive in demos and sandboxes but unable to enter controlled production, let alone scale. A common cause is not distinguishing evaluation goals across phases: pilot phase only looks at demo effects or local accuracy; production phase discovers reliability, unit economics, and bottom-line risks were never validated.

Ontology intelligence projects should not evaluate only at project closure.

Pilot phase: Focus on verifying scenario value signals, core capability questions, key safety boundaries, and technical feasibility;

Controlled production phase: Focus on real business effects, error distribution, action closure, operational stability, and unit task cost;

Scaling phase: Further evaluate cross-organizational reuse, marginal integration cost, operational capability, and whether risk worsens with scale.

The pilot phase gate answers "whether conditions for production entry are met"; this article's scorecard continuously judges after production entry: whether the project is still worth running and expanding.

Phase reviews must produce explicit decisions:

Expand: Mandatory thresholds passed, key quality metrics met, business value and unit economics validated → expand scope;

Time-boxed optimization: Value exists but some quality, cost, or operational metrics unmet → assign owner and re-evaluation date;

Restrict or rollback: Safety, action, or critical evidence issues → immediately shrink usage scope or revert subsequent requests to controlled version;

Stop: Long-term inability to prove business value, or sustained cost and risk exceed acceptable range.

Version rollback only changes handling of subsequent requests; it cannot undo already executed business side effects. For executed payments, limit adjustments, notifications, etc., reconciliation, controlled compensation, or manual handling must still follow business outcomes.

"Keep optimizing and see" is not a complete project decision. Any continued investment must bind specific gaps, target values, owners, and next evaluation date.

What an Evaluation Report Should Leave Behind

A single project evaluation should at minimum produce:

Clear business scenario, evaluation boundaries, and original baseline;

Mandatory thresholds and their verification evidence;

Metric definitions, targets, actuals, and trends for six dimensions;

Versions of ontology models, foundation models, data, rules, prompts, action contracts, agents, and runtime environment;

Result attribution, anomaly distribution, residual risks, and uncertainties;

Decision: expand, optimize, restrict, or stop;

Next-phase owners, resource commitments, and re-evaluation triggers.

Evaluation should establish baseline at project start, compare versions after each major release, conduct periodic retrospectives during production, and re-trigger when business rules, data sources, foundation models, or action boundaries change significantly.

Thus, the project scorecard becomes not a one-off table in a closure report, but the basis for management to continuously decide resource allocation and risk boundaries.

Summary

Ontology intelligence project success cannot be proven by concept count, triple scale, API count, or invocation frequency.

What truly needs evaluation: whether business results improved, whether semantics remain correct in real tasks, whether decisions are reliable, whether actions complete under control, whether safety boundaries stay effective, and whether unit task total cost is sustainable.

This series started from ontology semantics, then discussed responsibility layering, rules, actions, object states, capability questions, model release, layered acceptance, version operations, and project evaluation — forming a complete chain from "machine understands business" to "business capability runs continuously."

In the LLM era, enterprises easily focus on models, compute, tokens, and frameworks while neglecting business semantics, responsibility boundaries, and operational governance. This series aims not to create another technology cult, but a sustainable engineering order: align semantics with business, clarify responsibility through layering, connect systems with controlled actions, and evolve continuously with runtime feedback.

What ontology intelligence projects should truly accumulate is not more isolated concepts, triples, and fragile prompts, but verifiable, executable, governable enterprise intelligence assets that continuously create business value.