Why Ontology Projects Must Define Competency Questions First: Verifiable Acceptance from Day One

The previous article discussed the object runtime, which organizes current facts, derived states, time, versions, and evidence of concrete objects under a unified semantics, providing a traceable object projection for decisions and actions.

But this raises a more fundamental problem: a customer object may have hundreds of attributes and relate to many other objects such as contracts, receivables, disputes, collections, and tasks. Why does the current scenario require modeling these specific elements and not others?

Without explicit problem boundaries, ontology projects easily start by collecting terms, importing standards, and drawing concept diagrams. Concepts and relationships grow increasingly complex, yet the project cannot prove what business problems the model actually solves.

Competency Questions (CQs) in ontology engineering are designed to solve this problem.

Competency Questions are not about "what functions the system must perform," but rather predefine "which business questions this model must be able to answer to prove it has the expected capability."

"Competency Questions" Are Not About System Deficiencies

The term "Competency Questions" is abbreviated as "能力问题" (capability questions) in this article. This translation can be misunderstood as system incapability or missing agent tool capabilities.

In ontology engineering, competency questions refer to a set of natural-language questions that specify which queries, classifications, associations, inferences, or judgments the current ontology or semantic model should support. They define expected capabilities first, then derive model scope from those capabilities.

Classic competency questions mainly constrain the knowledge representation and reasoning scope of an ontology. This article further establishes traceability from competency questions to real data, rules, and runtime results, to verify whether the semantic model can enter enterprise systems — not to make the ontology responsible for data acquisition, rule execution, and business actions.

Several easily confused artifacts must be distinguished:

Value Narrative — Why do this, who it helps, what business outcomes it changes

Competency Questions — What questions the model must support answering

Semantic Model — What objects, relationships, states, rules, and action semantics are needed to answer those questions

Query, Rule & Runtime Implementation — How to obtain results based on model, data, and rules

Test Cases — How to verify expected results under specified inputs and exception boundaries

Business Metrics — Whether these capabilities create value in real work

Therefore, competency questions connect to the value narrative but are not project KPIs; they can guide query, rule, and test design, but should not be bound to a specific query language or database from the start.

"Define first" does not mean writing them once and never changing. A more reasonable approach is to establish an initial baseline before large-scale modeling, then continuously refine as business confirmation, data validation, and model implementation progress, while preserving the rationale and version for each change.

Why Define Before Modeling

If competency questions are written after the model is completed, they easily become post-hoc rationalizations: only selecting questions the model can already answer, avoiding critical questions the model cannot yet answer.

Defining competency questions before modeling brings at least four changes:

First, constrain model boundaries. No longer pursue "modeling all business knowledge into the ontology," but only model objects, relationships, states, rules, and evidence semantics required by current competency questions.

Second, expose data gaps. A question may be semantically expressible, but real data may not answer it. Mapping questions to real data earlier reveals issues such as object identifiers that cannot be linked across systems, key states not recorded, or rule versions not traceable.

Third, provide basis for modeling trade-offs. When a concept, relationship, or rule cannot be traced to a current competency question, it should be justified as a cross-scenario shared foundation, a regulatory or standard requirement, or deferred as over-modeling.

Fourth, shift acceptance from subjective review to repeatable verification. Experts still review whether concepts and relationships match the business, but acceptance no longer only checks if diagrams are complete and terminology is standard; it also uses real or controlled test data to verify that these questions yield expected results.

How to Write Good Competency Questions

"Can the system judge customer risk?" is not a sufficiently good competency question. It does not specify which object, at what time point, based on what facts and rules, what result to return, or how to answer when evidence is insufficient.

The following template narrows the question:

In [business scenario and time point] , for [business object] , can the system, based on [facts, relationships, rules, and evidence] , answer [specific business question] , return [result type and evidence] , and in cases of [missing, expired, or conflicting data] give [explicit boundary result] ?

For example:

During customer credit review, for a specified customer, can the system, based on current outstanding receivables, contract grace periods, accepted disputes, and applicable credit policies, determine whether the customer enters a credit warning state, return the judgment result, rule versions used, and evidence references, and return "insufficient evidence" when key data is missing or expired?

For questions that require repeated verification on similar objects, competency questions are usually written as parameterized question templates. Specific customer IDs, observation times, and receivable data belong in the test cases corresponding to that competency question.

This question can already constrain modeling scope, guide data preparation, query and rule design, and generate corresponding test cases.

Deriving Model and Data from a Set of Competency Questions

A scenario cannot have only a single overly macroscopic competency question. For the scenario "identify overdue risk and form credit disposal recommendations," it can be further decomposed into the following competency questions (CQs):

CQ-01: As of a specified time, what outstanding receivables does the customer have, and what are their uncollected amounts and overdue days?

CQ-02: Are these overdue receivables within contract grace periods, or are there accepted commercial disputes?

CQ-03: Under the currently effective credit policy, does the customer meet credit warning conditions, and what is the basis?

CQ-04: If the customer has entered a warning state, what disposal actions can be suggested, and which actions require manual confirmation?

CQ-05: If object identity cannot be linked, or key evidence is missing, expired, or conflicting, which judgments and actions should the system stop?

These five questions reverse-derive different requirements:

CQ-01

Required Model Elements: Customer, Receivable, Uncollected Amount, Due Date, Customer-Receivable Relationship

Real Data, Rules & Evidence: Customer ID mapping, due dates, receivable amounts, collection or write-off records, observation time

Verifiable Results: Receivable list, uncollected amounts, overdue days, and sources

CQ-02

Required Model Elements: Contract, Grace Period, Commercial Dispute, Acceptance Status, Object Relationships

Real Data, Rules & Evidence: Contract versions, clause effective times, dispute acceptance events

Verifiable Results: Grace period judgment per receivable, dispute acceptance status, and evidence

CQ-03

Required Model Elements: Credit Status, Policy, Applicability Conditions, Rule Semantics

Real Data, Rules & Evidence: Current business facts, policy version, rule execution results

Verifiable Results: Warning, no warning, or insufficient evidence, with basis

CQ-04

Required Model Elements: Action Semantics, Applicable Objects, Preconditions, Manual Confirmation Semantics

Real Data, Rules & Evidence: Disposal policies, action contracts, authorization conditions

Verifiable Results: Candidate actions, applicability conditions, confirmation requirements

CQ-05

Required Model Elements: Semantics of missing, expired, conflicting, and unknown states

Real Data, Rules & Evidence: Freshness markers, evidence completeness, degradation rules

Verifiable Results: Explicit indeterminate, suspended actions, or manual handling results

This forms a traceable chain:

Business Question → Competency Question → Model Elements → Real Data & Rules → Query or Inference → Expected Result → Business Value

For the object runtime discussed in the previous article, this chain also explains why the minimum object projection needs current facts such as receivables, grace periods, and disputes, version identifiers of applicable credit policies, and corresponding evidence references and freshness — not the customer's full set of attributes unrelated to the current problem.

Competency Questions Do Not Need to Be Answered by the Ontology Alone

"The model must support answering" does not equal "all results must be derived by OWL reasoning."

Different competency questions may be answered by different mechanisms working together:

Object types, relationships, and classification conclusions can be supported by the ontology and reasoning mechanisms;

Current uncollected amounts, overdue days, and dispute statuses need to be provided by business systems, or organized by the object runtime as current facts with time and evidence;

Credit warnings and disposal strategies may be executed by decision tables, rule engines, or business code;

Dispute reasons and complex contexts in text can be assisted by large language models, but evidence and manual confirmation boundaries must be retained;

Candidate actions can be formed by decision rules combined with action contracts, while actual execution remains the responsibility of controlled business services.

Therefore, competency questions verify whether business questions can be reliably answered under a unified semantics, not whether a particular ontology language can handle the entire chain.

How to Form a Usable Competency Question Baseline

Competency questions should not be written solely by ontology engineers in a meeting room. They should come from already selected business scenarios and real decision points, jointly created by business experts, Forward Deployed Engineers (FDEs) or knowledge engineers, data personnel, and development/test staff.

The process can advance in six steps:

Select a key node requiring judgment, explanation, or action from the value narrative and business process.

Let the business person actually responsible for that node raise the real questions they must answer — do not start by asking what features they want.

Narrow each question using object, scenario, time, basis, result, and exception boundaries.

Establish traceability for each question to model elements, real data, rules, evidence, and business value.

Prepare test scenarios for positive, negative, data missing, expired timeliness, and factual conflicts, and specify expected results.

Have the business owner confirm question semantics and expected results, forming a versionable competency question baseline.

This baseline should not consist only of question text. Each competency question must also associate business scenario, responsible role, priority, expected result type, model element mapping, data availability, test cases, and current verification status.

From an engineering perspective, this resembles Test-Driven Development (TDD) and Behavior-Driven Development (BDD) in software engineering, and can be understood as Test-Driven Ontology Development (TDOD) : first define the verifiable behaviors the model and its runtime implementation must exhibit, then model and implement. However, competency questions are only natural-language acceptance intentions, not automated test scripts themselves.

During implementation, a competency question typically requires multiple test scenarios. Parts suitable for automation can be implemented as SPARQL query assertions, SHACL constraint tests, rule engine test cases, or semantic API tests; if the underlying store is a property graph, Cypher or Gremlin assertions can be used. Incorporating these automated tests into the model CI/CD pipeline as pre-release regression gates allows timely detection of capability regression when concepts, mappings, or rules change.

Competency Questions Are the Acceptance Entry Point, Not the Endpoint

A set of competency questions passing under specified model, data, and rule versions and test boundaries only proves the current model and runtime implementation support those questions. It cannot alone prove:

Concepts, relationships, and constraints in the full model not covered by these competency questions are equally correct;

Real data is always complete and trustworthy across a larger scope;

All rules, exception paths, and action contracts are fully covered;

Agents execute stably under real traffic;

The project has created the expected business value.

Therefore, competency questions are important inputs for modeling scope and model acceptance, but not a complete acceptance system. Subsequent checks must cover real data, semantic constraints, rules and exception paths, action contracts, evidence traceability, and business outcomes.

They also do not equal project evaluation. The model answering questions does not guarantee business users will adopt it, nor that efficiency, risk, and cost metrics have improved.

Summary

The value of competency questions lies in defining before modeling what questions this model must support answering, and tracing business questions to model elements, real data, rules, evidence, expected results, and business value.

They shift ontology projects from proving completion by concept count, triple scale, or expert impression, to constraining modeling scope and engineering delivery with a set of repeatably verifiable business questions.

Once the relationships among competency questions, model elements, real data, and expected results are clear, the next step is to publish the business-confirmed model as an engineering baseline and transform it into engineering artifacts that runtime systems can use and verify.

Competency questions are not a question list to decorate documentation after the model is built; they are the verifiable requirements baseline that an ontology project should establish before modeling begins.