Skip to content
Ontology Evaluation: When to Continue, Scale Back, or Stop Building
R&D Management 22 min read

Ontology Evaluation: When to Continue, Scale Back, or Stop Building

This article provides a decision framework for ontology projects, distinguishing goal-constrained and open-exploration evaluation modes, using structured comparison tables to separate errors from unknowns, correctly attributing improvements, accounting for maintenance costs, and guiding staged decisions (continue, reduce, pause, archive) based on evidence rather than model size.

Data Bricklaying Diary
Data Bricklaying Diary
Data Bricklaying Diary

Opening Scenario

An equipment ontology has been in use for months. Concepts have grown, more data sources are connected, and queries are faster. Yet the manager asks: "How many fewer phone calls do schedulers make? How many fewer return trips?" The team cannot answer. The model has been refined, but no one has seriously compared whether the business actually became easier.

Two Construction Modes, Two Evaluation Lenses

The previous article checked whether modifications were confirmed, settled, and used. Now the question is: what actually improved, and is it worth continuing? The answer depends on the construction mode:

Goal-constrained : Did we solve the agreed problems, and at what cost?

Open exploration : What did this round clarify? Is the evidence sufficient? What is worth investigating next?

Model Growth ≠ Maturity

Adding a department may add a batch of objects and relations; quantity indicates scope change, not usefulness. A model unchanged for a long time may mean definitions are sufficient, or that the business side stopped raising issues. If no errors reported for a month, first ask whether anyone is still using it and whether there is a channel to report problems.

Separating Judgment Criteria

The author separates four distinct questions, each with its own evidence and limitations:

What to judge                | Evidence to examine                              | What this cannot prove
-----------------------------|--------------------------------------------------|-----------------------------------
Can required meanings be
expressed?                   | Check definitions, relations, examples per
                             | problem                                          | Expressible ≠ data complete
Are definitions clear and
checks passed?               | Semantic review records, model/data check
                             | reports                                          | Checks passed ≠ real records accurate
Is processing per agreement
during use?                  | Manual usage records, query/filter/output
                             | case reviews                                     | Compliance ≠ task completed
Did the business benefit?    | Verify hours, rework, delivery times, business
                             | records                                          | Local improvement ≠ all from ontology
Two orientations, different gains
Two orientations, different gains

Even before a system goes live, you can observe how people use definitions. For example, purchasers and maintenance staff using the same definitions to verify equipment: can they now explain distinctions in one go that previously required repeated explanations? Record such usage. If the model was just handed over and no one has used it, mark effect as "not yet evaluated" — do not rush to calculate headcount savings.

These evidences must not be merged into a single score. Faster queries cannot offset recommending equipment that is under repair; complete definitions cannot offset missing data.

Compare Lists, Don't Just Report Accuracy

Using the repair-status filter as a running example: when equipment identity and source are confirmed, records are valid, and no unresolved conflicts exist, "under repair" should be excluded and "not under repair" passes this item. If identity/source unconfirmed, status missing/expired, or unresolved conflicts exist, the case goes to pending verification. Passing this item still does not equal dispatch approval.

A batch of 40 cases illustrates the comparison. Assume 10 should be excluded, 20 should pass, 10 should be pending. Old and new programs are compared with identical inputs, query time, and criteria.

Comparison item                              | Old handling | New handling
---------------------------------------------|--------------|-------------
Of 10 should-exclude, erroneously passed     | 3/10         | 0/10
Of 20 should-pass, erroneously excluded      | 1/20         | 1/20
Of 10 pending, correctly marked pending      | 4/10         | 10/10
40 cases, three result types counted separately
40 cases, three result types counted separately

The remaining 7 should-exclude cases were correctly excluded in old, all 10 in new; 19 should-pass cases passed both. Old treated 6 unknown cases as pass; new marked all 10 pending. Full output distribution must be reported — no hiding results outside the table.

In this batch, erroneous passes and unknown misjudgments decreased, but erroneous exclusions persist. The result can only be stated here; we cannot claim filtering is fully correct or that equipment is safe for dispatch.

Denominators must match the problem: false inclusion denominator = should-exclude cases; false exclusion denominator = should-pass cases; unknown handled separately. Reporting only "how many in the returned list are correct" misses devices wrongly blocked.

Missing samples = uncovered; denominator zero = not applicable. Do not write "zero errors" for "not tested". Key risks must be examined individually, not averaged away by large numbers of simple cases.

Sample selection must be explained: include reproduced original errors, normal controls, different sources, confusing cases, and retain non-debug materials. Dozens of hand-picked cases only show this check's results, not the overall business error rate.

Data and Methods Both Changed — Who Gets Credit?

Previous articles found one cause: the ontology expressed the repair-exclusion requirement, but the query didn't use it. Only the query was fixed, so the improvement belongs to the query fix, not to ontology expansion.

If repair data was added, extraction model changed, and definitions adjusted simultaneously, the result may be better but contributions cannot be disentangled.

To isolate an effect, use the same materials and inputs in an isolated environment, changing only one thing. If multiple changes are necessary, report the whole package's change and note that individual contributions cannot be separated.

Multiple changes together, don't credit ontology alone
Multiple changes together, don't credit ontology alone

Controlled comparison uses fixed inputs for processing differences; field observation uses current rules and valid data. Do not use expired statuses or invalid rules in real dispatch just to keep conditions comparable.

Business before/after comparison must also consider task mix changes. Previously only fully-documented owned equipment; later added leased equipment and missing records — average time up may not be regression. Conversely, less business or more skilled staff may speed delivery.

Compare similar tasks, retain source, time range, version, sample size, and differences. State unexcludable influences explicitly; do not rush to attribute changes to the ontology.

Pending Verification Increased — Worse or More Honest?

Previously no status meant normal; now marked pending. Auto-pass rate may drop, manual work increases. This may expose hidden risks. But "safer" cannot justify everything: if all cases go manual, the business becomes unusable.

Must check: did verified errors decrease? Why sent to manual? Is anyone picking them up? Do waits delay delivery? Cases that should pass but were sent to manual — find causes.

Long-term unknowns may indicate identity mapping or data supply unsolved. The next round may need to fix sources and handoffs; adding concepts may not help.

Measure effect from the scheduler's daily work: time to verify a list, people contacted, rework from erroneous dispatch, on-time delivery. Define clock start/end; separate active work hours from elapsed time. Three days elapsed may be only one hour of active work waiting for status confirmation.

Issue handling cycle: closed issues from discovery to re-verification; open issues track wait time. Counting only easily closed tickets hides stuck ones.

Saved Time Must Be Viewed with New Work

Previously schedulers called per device; now query returns list in seconds. Cannot count all verification time as saved.

New process may still need unknown confirmation, plus expert review, data maintenance, program checks, release verification. Work is distributed across roles, easily missed — especially expert time.

Cost accounting: include initial modeling, data integration, daily verification, expert confirmation, tool operation, later modifications, migrations, caller adaptations. Reference existing project records.

Separate first-time vs ongoing maintenance; compare over same period. Pilot-end projections are estimates; actual reduced work observed is real result.

Saved work and new work together
Saved work and new work together

Saved hours ≠ financial savings. If time used for more tasks, report capacity/staffing changes; without evidence, do not convert to realized cash. Do not double-count "query speedup" and "rework reduction" in the same period.

Risk control value may not monetize short-term. Record which errors caught, which risks remain uncovered; judge necessity per business requirements. No accidents does not prove the model prevented them; cannot infer risk control valueless.

Open Exploration: Judging This Round's Gains

Equipment filtering has clear criteria. Sentiment analysis may shift topics (brand today, celebrity tomorrow) — cannot demand a fixed business endpoint.

But this round can still ask specifics: checking opinion attribution, discussion objects, event merging? What judgments are supported, which explanations excluded, what to verify next?

Example: previously counted retweeted negative speech as publisher's negative review. Improved: check confirmed-attribution materials for reduced misjudgment, also check if direct negative expressions missed. Unjudgeable stance → keep unknown.

Brand vs celebrity materials separate. Previously only brand representation, now also figures → scope expanded. Original materials' attribution more accurate → separate improvement. New topics increase nodes/hits → cannot claim original capability improved.

Valuable exploration outcome can be: verifying existing model sufficient (no extension needed), or discovering two same-title events lack merge basis.

Phase record table:

What this round clarified                          | Evidence & limitations                          | How affects next round
---------------------------------------------------|-------------------------------------------------|-----------------------------------
Material only quotes others' negative review;
publisher says no experience, no evaluation        | Keep original text; cannot count as publisher's
                                                   | negative review; true stance still unknown      | Use another batch to test if
                                                   |                                                 | direct vs retweet distinction holds
Two same-title materials cannot merge as one event | Date and context insufficient; keep separate    | If merge-supporting materials
                                                   |                                                 | obtained, verify; else keep undecided
New topic temporarily needs no new concepts        | This round's examples expressible with existing
                                                   | definitions; other materials not yet covered    | Reuse first; propose changes only
                                                   |                                                 | when concrete expression gaps appear

These conclusions may not immediately save money or people, but they clarify how far this round reached. Next round's target explanations and missing materials are more concrete than "continue enriching knowledge base". Many questions ≠ achievement; long-unsupported ones need not drag on.

Research pivot: record original questions, why pivot, next questions, planned investment. Old questions paused → write deferred/unresolved; don't treat new findings as old tasks done. Also check if original capabilities broken by changes — separate from whether new direction worth pursuing.

The series' minimum runnable example only checked pre-structured input program behavior. Inputs, expectations, programs prepared by author; no validation of auto-extraction or real business benefits. It helps discuss checking methods, cannot substitute for the above evaluation.

At Evaluation Node, Decide Next Investment

When to evaluate: delivery nodes, business cycles, risk changes — not wait for "model complete". Look at: what to solve next, what evidence obtainable, how much more to spend — not continue just because already invested heavily.

Decision table:

Stage decision   | When to consider                                                          | What to communicate next
-----------------|---------------------------------------------------------------------------|-----------------------------------
Continue         | Existing improvement worth keeping/expanding, or important questions
                 | remain verifiable with a checkable next step                              | Next round questions, material
                 |                                                                           | conditions, investment cap, owner,
                 |                                                                           | review point
Reduce scope     | Some uses effective, but other sources/scenarios lack conditions long-term| Which capabilities kept, which
                 |                                                                           | promises stopped, how users
                 |                                                                           | transition
Pause expansion  | Key data, confirmers, or authorization temporarily unavailable; modeling
                 | hard to advance                                                           | Wait conditions, current limits,
                 |                                                                           | handover person, re-eval time
Archive/close    | This round's agreements completed, or confirmed termination of this work  | Version, evidence, unfinished
                 |                                                                           | items, restart conditions; running
                 |                                                                           | dependencies handled separately
Continue, reduce, pause, archive
Continue, reduce, pause, archive

Exploration phase not yet proven business value can still invest another round, provided you can articulate why this round is worth investigating, to what depth, and max time/resources. E.g., limit to obtainable materials, test if new topics reuse existing definitions; at agreed node if insufficient evidence, reduce or pause — not "add more data" as infinite renewal.

Decision to stop expansion cannot abandon running services. Model archived but others may use. Promised items, unfinished impacts, necessary maintenance must be handed over. Project terminated, unfinished still written as unfinished.

Finally, record decision back to project record: what materials/versions compared, what found, what still unknown, actual cost, next steps. For continued investment, specify scope, max investment, review date, conditions for reduce/stop. Dissenting opinions recorded, no forced aggregate score.

Manager asked "fewer calls, fewer return trips" — if not measured, say not measured. Next, pick one truly worth measuring, agree how to record, when to review. Better than reporting another batch of new concepts to justify next investment.

Original Source

Signed-in readers can open the original source through BestHub's protected redirect.

Sign in to view source
Republication Notice

This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactadmin@besthub.devand we will review it promptly.

R&D managementknowledge engineeringcost-benefit analysisontology evaluationevidence-based decision makingproject assessment
Reader feedback

How this landed with the community

Sign in to like

Rate this article

Was this worth your time?

Sign in to rate
Discussion

0 Comments

Thoughtful readers leave field notes, pushback, and hard-won operational detail here.