Why Your RAG System Still Gets Business Logic Wrong (And When Ontologies Help)

The Problem: Correct Policy, Wrong Answer

A delivery assistant retrieves the correct policy — "equipment counts as delivered only after customer acceptance" — and queries the ERP for "completed" orders. The resulting list misses several orders because the ERP's "completed" flag only means the outbound process finished; some equipment is still being installed or awaiting acceptance.

Swapping in a stronger model or adding an ontology won't help until you pinpoint where the assistant conflated "shipped" with "accepted".

Don't Blame RAG Alone

RAG (Retrieval-Augmented Generation) combines retrieval with generation; it can filter by version, use keywords, and employ agentic multi-step retrieval (Lewis et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, NeurIPS 2020, https://arxiv.org/abs/2005.11401; Microsoft Azure Architecture Center, Design and develop a RAG solution, https://learn.microsoft.com/en-us/azure/architecture/ai-ml/guide/rag/rag-solution-design-and-evaluation-guide). RAG and ontology serve different roles: RAG fetches relevant material, while an ontology explicitly models business concepts, relationships, and constraints for both humans and programs to share.

Tracing a Single Wrong Order

The assistant must stitch together data from ERP (shipment), service system (installation), and project-maintained acceptance records. The errors fall into distinct categories:

Policy not found or outdated version — check indexing, retrieval strategy, versioning, and applicability. Ontology can supply terminology and applicability links, but indexing still needs maintenance.

Mistaking "shipped" for "accepted" — examine how the assistant interprets fields and what mappings the query uses. Ontology makes the distinction between shipment, installation, and acceptance explicit for shared use.

Missing acceptance records or unlinked to orders — verify data entry, interfaces, object identifiers, and linkage rules. Ontology specifies which facts are required but cannot create missing facts.

Batch delivery miscalculated or logic errors — review calculation rules, reasoning process, and program implementation. Ontology describes objects and constraints, but computation still requires implementation and testing.

Query results cannot modify business state — confirm permissions, approvals, and business APIs. Ontology can express action conditions but cannot replace authorization and actual writes.

A single list may contain omissions, false positives, and items needing verification. Fix the mapping first; then supplement records. Lumping everything as "model doesn't understand business" obscures the real fix.

When to Model Business Definitions Separately

If only one assistant, one contract type, and a few fixed fields exist, clarifying definitions, correcting queries, and adding tests may suffice. Trouble arises when multiple applications — delivery dashboards, customer-service bots, sales forecasts, escalation workflows — each implement their own "delivery completed" logic. A change (e.g., new batch-acceptance clause) then requires coordinated updates across teams.

Ask: can we manage the shared business definition in one place, recording version applicability and effective dates? If an existing delivery-state service already governs the definition, reuse it. If definitions are scattered across code, prompts, and documents, consider a shared business model that people can review and programs can consume.

For delivery, the concrete questions are:

Is delivery judged per order, per device, or per contract-defined batch?

Which valid acceptance record proves completion, and which devices or batches does it cover?

What step does each field in ERP and the service system represent?

Which contract types use this definition, when does it take effect, and who confirms it?

After definitions are confirmed, they must be mapped to real data; publishing the model does not auto-sync data or fix old queries. Maintenance ownership — who revises definitions, who validates mappings, which applications to re-check — must be assigned.

RAG and Ontology Working Together in a Query

For the manager's request — "orders due by month-end that are not yet delivered" — a combined flow could be:

Clarify organization and cutoff date; "month-end" maps to a concrete date.

RAG retrieves applicable policies and contract clauses for the assistant to cite. Special contracts not yet interpreted are flagged for verification, not forced into the general rule.

The model structures a query with organization, due-date range, and fact-as-of timestamp. The application validates parameters and permissions, then hands off to a semantic query service that executes against confirmed business definitions (using SQL, graph queries, or existing business APIs — no mandatory graph database).

The query service joins orders, devices, batches, and acceptance records using the published mappings. Current facts come from authorized business systems or fresh-enough replicas, not stale reports.

The ontology definition serves as context for the model, but the query service must use the published mappings, and the rule service must implement and test the corresponding logic. Newly retrieved clauses cannot override production rules without business sign-off.

The rule service filters devices/batches due by month-end, checks acceptance status, and outputs separate lists of pending items and items needing verification. Batches due next month stay out of this month's list.

The LLM explains why each item was included and what evidence is missing; the order list and statuses are rendered programmatically to avoid omissions or hallucinated rewrites.

Producing a list does not grant authority to change state. "Not yet delivered by month-end" is not "overdue". Any state change requires fact verification, permissions, and approval — not a direct write from the query result.

Fair Evaluation: Same Orders, Same Rules

First fix the obvious retrieval, data, and program bugs in the existing solution. Then compare it against the ontology-enhanced approach using the same confirmed business definitions, same materials, same rule versions, same permissions, and same fact cutoff. Have business experts verify a batch of orders (including normal completions, shipped-but-not-accepted, batch acceptance, missing records, and contract exceptions) and hold out a test subset.

Measure three things:

Result accuracy — which pending items were missed, which completed items were falsely reported, and whether insufficient-evidence cases were properly flagged. The system must still decide automatically when evidence is sufficient.

Caliber consistency — under the same definition, version, data timestamp, and permission scope, dashboards and assistants should return identical results.

Maintenance cost — when delivery conditions change, how many places need updates, which mappings must be re-validated, and how much expert time is consumed.

If the improvement mainly comes from entering missing acceptance records or correcting queries, credit those efforts. The ontology is worth expanding only if it additionally reduces errors, duplicate explanations, and maintenance work — not just because a demo looks smoother.

Conclusion

Returning to the incomplete list, the real question is: "At which step did it misuse the business meaning?" Once identified, you know whether to fix data, fix code, or extract repeatedly confused definitions into a shared model. Whether today's modeling effort becomes obsolete as LLMs improve will be examined next: which practices may be discarded and which business knowledge is worth keeping.

References

Lewis et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks , NeurIPS 2020. Explains the basic RAG approach. https://arxiv.org/abs/2005.11401

Microsoft Azure Architecture Center, Design and develop a RAG solution . Covers retrieval strategies, evaluation, and Agentic RAG extensions; the delivery case and comparison method are the author's design. https://learn.microsoft.com/en-us/azure/architecture/ai-ml/guide/rag/rag-solution-design-and-evaluation-guide

W3C, OWL 2 Web Ontology Language Document Overview (Second Edition) . Describes OWL's capacity to express concepts, properties, and relationships; not a mandate for enterprise adoption. https://www.w3.org/TR/owl2-overview/