Databricks Genie Ontology: Solving Metric Version Conflicts in AI Agents

Background: The Metric Version Problem

Databricks and Turing created a fictional UK energy supplier, Calderwick Energy, to test three Data Agent systems. One test question asked: "What was the Contract for Difference (CFD) plan rate set by the commercial department for the 2024/25 fiscal year?"

The billing cost table contained a field cfd_rate_p. Filtering for fiscal year 2024/25 returned 0.31 pence/kWh consistently across rows. Two benchmark agents ran the query five times each and always returned 0.31, missing the correct answer.

The explanation lay in the Notebook that generated the table. The commercial department sets plan rates annually. The billing cost table used the 2025/26 rate of 0.31 pence/kWh as an assumption, applying it retroactively to earlier periods. The Notebook preserved historical versions: 0.52 pence/kWh for 2024/25 (the actual plan rate set that year) and a draft 0.45 pence/kWh for 2026/27.

0.52 pence/kWh — Fiscal year 2024/25 — Plan rate set by commercial department that year

0.31 pence/kWh — Fiscal year 2025/26 — Rate assumption used in billing cost table, also applied to earlier periods

0.45 pence/kWh — Fiscal year 2026/27 — Draft rate for next fiscal year

If the business asks "What rate was used to calculate this batch of bills?" 0.31 is correct. If they ask "What rate did the commercial department set that year?" the answer is 0.52. SQL's fiscal-year filter locks the billing period, but the rate version must be confirmed from source materials. Genie answered 0.52 correctly in all five runs, first retrieving an ontology snippet containing historical rates, then opening the original Notebook to verify.

Genie Ontology: Two Types of Reusable Knowledge

Enterprises often record rules in disparate places: data tables store computed results, BI semantic models store formulas, Notebooks explain why a formula was used. Analysts know where to look; a new Agent may not navigate the correct path even with access to all materials.

Genie Ontology splits knowledge into two categories:

Learned knowledge (auto-extracted): Genie extracts calculation logic, business rules, and data relationships from dashboards, SQL, Notebooks, and (via Power BI integration) semantic model measures, calculated columns, and relationships. These become Ontology Snippets reusable across questions. In the rate example, a snippet linked historical plan rates to the source Notebook, enabling Genie to trace the evidence.

Modeled knowledge (explicitly governed): In Unity Catalog, teams define metrics via Metric Views , organize department knowledge via Domains , and record authoritative concept explanations via Pages . For net revenue, the calculation logic goes in a Metric View; applicable reports, business scope, and version timestamps go in the business definition.

These two types carry different governance costs. Core definitions (financial close, executive reporting) warrant explicit owner sign-off. Ad-hoc analyses and department-specific rules can be auto-extracted, with formal definitions added when high-frequency ambiguity arises. When issues occur, data teams can trace a Genie answer to the snippet used, inspect source assets, authors, and lineage, then decide whether to revise the source or promote the concept to formal governance.

Handling Multiple Concurrent Versions: Authority and Applicability

Connecting more sources surfaces disagreements previously isolated in different systems. A common report may use the current definition, a historical review uses the period's definition, and a draft exists for next quarter — all labeled with the same metric name.

Databricks' OntoRank ranks candidate snippets using signals: source authority, certification status, author relevance to the question, usage frequency, and freshness. Official documentation states that within the same topic, explicitly modeled knowledge, certified assets, and human-curated Genie Agents generally outrank auto-extracted knowledge. Remaining conflicts may trigger a clarifying question to the user.

For data governance practitioners, the rate case highlights a key requirement: record both the asset update time and the rule effective time separately. The billing cost table may update daily, yet its 0.31 value always reflects the calculation-time assumption. An untouched Notebook from years ago may remain the best evidence for historical decisions. Certifying a business definition requires not just the formula but also applicable business scope, effective date, draft status, and the team responsible for interpretation and changes. Otherwise, an Agent may apply a certified asset to the wrong period or scenario.

Access scope also affects who can use definitions. Internally derived snippets respect Unity Catalog permissions. External sources like Power BI (currently Beta) make snippets available to all users in the same Databricks account. Enterprises onboarding external semantic models must review which business definitions are shareable under this model.

Benchmark: 91% Pass Rate Across Three Full Systems

Databricks and Turing ran tests in a simulated enterprise workspace containing hundreds of tables, expired documents, deprecated dashboards, and development copies. Three systems received identical questions; each ran five times per question; answers were judged against predefined rules.

Genie — Task Pass Rate: 91% — Avg Runtime: 100.3 sec — Avg Tokens: 431,829

Anonymous Agent A — Task Pass Rate: 45% — Avg Runtime: 434.6 sec — Avg Tokens: 889,019

Anonymous Agent B — Task Pass Rate: 58% — Avg Runtime: 306.5 sec — Avg Tokens: 1,051,070

Genie's average token consumption per run was less than half of either baseline agent. The rate question illustrated a path difference: following tables and fields repeatedly returned 0.31; with a snippet pointing to historical plan rates, Genie went straight to the Notebook for verification. Different evidence paths mean different tool calls and search times.

Important context: Genie runs natively; the other two agents accessed Databricks via the Databricks toolset, which did not include Genie MCP. The 91% figure is this system's pass rate on this specific benchmark.

To estimate the value of business context in their own environment, enterprises should use the same model, same tools, and same question set, comparing runs with governance context enabled vs. disabled, while tracking pass rate, latency, and token usage. This isolates whether gains come from table discovery, definition selection, or evidence verification.

Validating Data Agents: Test One Metric Across Four Versions

Pick a metric that has genuinely changed definition and is frequently asked by leadership. Using the same dataset, pose four questions:

What is the result under the current definition?

What was the officially published number for the prior year?

What would the prior year's result be if recalculated with the current definition?

Has the next draft definition taken effect?

All four may query the same tables but require different evidence. During validation, examine both answer and evidence: which version did the Agent choose, why is it applicable, and can a business user trace the citation? Then remove a key artifact and observe whether the Agent asks for clarification, states missing evidence, or substitutes another version. The first test checks correctness when evidence is complete; the second checks behavior when evidence is missing.

For enterprises deploying query agents to operations or finance, the highest-ROI investment is on metrics where a wrong answer changes decisions and multiple versions already exist. Assign clear owners, retain effective dates and original sources for each version, and add confusing cases to pre-release regression tests. Then, when someone presents 0.31 and 0.52, the system can explain each number's origin and which one applies to the current meeting.

References

Databricks test and rate case study

Genie Ontology release notes · Databricks product documentation

Genie conflict handling and permissions documentation · External knowledge source documentation

Databricks test methodology and results