Databricks Genie Ontology: Why Two AI Agents Failed on One Business Metric

Background: Databricks Genie Ontology Launch

On November 11, Databricks announced general availability of Genie Ontology. The same day, a public test case revealed a core challenge: when the same metric exists in multiple versions across reports, SQL, and Notebooks, an agent must decide which version applies to the user's question and let business users trace the evidence.

The Core Problem: One Metric, Multiple Versions

Databricks and Turing created a fictional UK energy supplier, Calderwick Energy, to test three Data Agent systems. The test question asked: "What was the Contract for Difference (CFD) plan rate set by the commercial department for the 2024/25 financial year?"

The billing cost table contained a field cfd_rate_p showing 0.31 pence/kWh for 2024/25. However, the Notebook that generated the table revealed the commercial department originally set 0.52 pence/kWh for 2024/25, while 0.31 was the 2025/26 rate applied retrospectively. A draft 0.45 existed for 2026/27.

0.52 pence/kWh – 2024/25 – Commercial department's original plan rate

0.31 pence/kWh – 2025/26 – Billing table's rate assumption, also used for earlier periods

0.45 pence/kWh – 2026/27 – Draft rate for next financial year

Both comparison agents (Anonymous Agent A and B) repeatedly returned 0.31 because they followed the table and filter. Genie, in five runs, returned 0.52 on the first attempt by retrieving an ontology snippet pointing to the historical plan rate and then opening the source Notebook to verify.

Genie Ontology: Two Types of Reusable Knowledge

Genie Ontology organizes knowledge into two categories:

Learned Knowledge (Ontology Snippets)

Automatically extracted from existing assets—dashboards, SQL queries, Notebooks, and Power BI semantic models. These snippets capture calculation logic, business rules, and data relationships. For the CFD rate, a snippet linked the historical plan rate to the original Notebook, enabling Genie to trace the evidence.

Modeled Knowledge (Explicit Definitions)

Curated and maintained by teams in Unity Catalog using Metric Views (calculation logic), Domains (departmental organization), and Pages (authoritative business concept explanations). For example, net income calculation logic goes in a Metric View, while applicable reports, business scope, and version history go in a Page.

The two types carry different governance costs. Core metrics like financial settlements warrant explicit modeling by business owners. Ad-hoc analyses can rely on extracted snippets, with frequent ambiguities later promoted to formal definitions. When issues arise, data teams can trace a Genie answer to the snippet, inspect source assets, authors, and lineage, then decide whether to revise the source or promote the concept to governed knowledge.

Handling Conflicts: Authority, Scope, and Time

When multiple assets define the same metric differently—e.g., a current report uses the latest definition, a historical review uses the period's definition, and a draft exists for next quarter—Genie's OntoRank ranks candidates using signals: source authority, certification status, author relevance to the question, usage frequency, and freshness. Official documentation states that explicitly modeled knowledge, certified assets, and human-curated Genie Agents generally outrank auto-extracted knowledge. Remaining conflicts may trigger a clarifying question to the user.

A key governance insight: the update timestamp of an asset and the effective date of the rule it encodes must be recorded separately. The billing table updates daily but its 0.31 value always reflects the calculation-time assumption. An old, unchanged Notebook may remain the best evidence for historical decisions. Certifying a business definition requires not just the formula but also applicable business scope, effective date, draft status, and the responsible team. Without these, an Agent might apply a certified asset to the wrong period or scenario.

Access scope also matters: snippets derived from internal assets respect Unity Catalog permissions. External sources like Power BI (currently Beta) make their snippets available to all users in the same Databricks account, so enterprises must review which business definitions are safe to share.

Benchmark Results: 91% Pass Rate with Half the Tokens

The test ran in a shared workspace containing hundreds of tables, expired documents, deprecated dashboards, and development copies. Each system received the same questions and ran each five times. Results:

Genie : 91% task pass rate, 100.3 seconds average runtime, 431,829 average tokens

Anonymous Agent A : 45% pass rate, 434.6 seconds, 889,019 tokens

Anonymous Agent B : 58% pass rate, 306.5 seconds, 1,051,070 tokens

Genie's average token consumption was less than half of either comparison agent. The CFD rate question illustrates why: without a guiding snippet, agents repeatedly scan the table and get 0.31; with a snippet pointing to the historical rate, Genie goes straight to the Notebook. Different paths mean different tool calls and evidence-gathering time.

Important caveat: Genie runs natively, while the other two agents access Databricks via the toolset without Genie MCP. The 91% is this system's pass rate on this benchmark. To measure the value of business context in their own environment, enterprises should run the same model, tools, and questions with and without governance context enabled, tracking pass rate, latency, and token usage to pinpoint where gains come from—table discovery, definition selection, or evidence verification.

Validating Data Agents: Test One Metric Across Four Versions

For enterprises rolling out query agents to operations or finance, the article recommends a concrete acceptance test. Pick a metric that has changed definitions and is frequently asked by leadership. Using the same data, ask four questions:

What is the result under the current definition?

What was the published figure for the prior year?

What would the prior year yield if recalculated with the current definition?

Has the next draft definition taken effect?

All four may hit the same tables but require different evidence. During validation, examine both answer and evidence: which version the agent chose, why it applies, and whether a business user can follow the citation to verify. Then remove a key artifact and observe whether the agent asks for clarification, states missing evidence, or silently substitutes another version. The first test checks correctness when evidence is complete; the second checks behavior when evidence is missing.

Priority should go to metrics where a wrong answer changes decisions and where multiple versions already exist. Assign clear owners, preserve effective dates and original sources for each version, and add confusing cases to regression tests for every release. Then, when someone presents 0.31 and 0.52, the system can explain the origin of each and which one belongs in the current meeting.

References

Databricks test and CFD rate case

Genie Ontology release notes · Databricks product documentation

Genie conflict handling and permissions documentation · External knowledge source documentation

Databricks test methodology and results