Databricks Genie Ontology: Mining Business Context from SQL and Usage Patterns

Core Problem: Business Meaning Lives in Behavior, Not Just Schema

The hardest enterprise data challenge is not complex SQL but semantic ambiguity: which table defines Revenue — orders or financials? Do refunds count? Natural month or fiscal month? When two dashboards conflict, who wins? These rules are rarely fully captured in schema or data dictionaries.

Traditional Approach: Explicit Knowledge Engineering

Classic semantic layers require upfront manual definition of metrics, entities, synonyms, and governance signals (certification, deprecation). Databricks Unity Catalog Semantics provides this: Metric Views, Domains, Pages, Certification, Deprecation. Reliable but cannot cover the long tail of tribal knowledge.

Genie Ontology's Dual-Layer Architecture

Genie Ontology acts as a unified context layer for the Genie AI agent, merging two sources:

Governed Semantics (Unity Catalog) : Human-curated Metric Views, Domains, Pages, Certification, Deprecation — deterministic, high-priority definitions.

Inferred Context : Automatically extracted Ontology Snippets from existing assets and usage — Metric Views, Dashboards, SQL Queries, Genie Agent interactions.

A Snippet can be a metric definition ("Active User = cross-platform deduplicated users"), a data-source preference ("Revenue questions should use the finance team's certified Genie Agent"), or a business rule ("Only leads with a booked demo count as Qualified Leads").

From Knowledge Engineering to Context Mining

SQL is not just query code; it encodes business judgment: table selection, join logic, status filters, time-field choice, aggregation granularity. Dashboards signal which queries become long-lived analytical assets. Usage history reveals which assets remain active versus one-off experiments. Genie Ontology treats these execution traces as first-class context sources — shifting from "how should the company define this?" to "how has the company actually been using this?"

Authority Scoring: Resolving Conflicts Among Multiple Definitions

When Genie discovers multiple versions of a concept (e.g., three Revenue algorithms: an ad-hoc SQL from two years ago, a widely used sales dashboard, a certified finance Metric View), vector similarity alone cannot decide which to trust. Genie assigns an authority score based on:

Source type (certified Metric View > dashboard > ad-hoc query)

Usage frequency and recency

Expert authorship and expert consumption

Asset popularity in the usage graph

Databricks builds a usage graph from asset–usage relationships and propagates authority via a PageRank-like algorithm, weighting expert authorship, expert consumption, and asset popularity. At query time, relevant snippets are retrieved, ranked by authority, conflicts are resolved, and a task-specific context is assembled.

Not Just a Bigger RAG: Governance, Freshness, Permissions

Standard RAG answers "which text is most similar to the query?" Enterprise agents need "which definition is currently organizationally endorsed?" Genie Ontology layers authority, governance state (certified/deprecated), freshness, usage signals, and Unity Catalog permissions on top of retrieval. Snippets inherit permissions from their source assets, preventing the context layer from becoming a permission bypass.

Execution Flow: Context Resolution Before Computation

The chain: Model understands intent → Ontology narrows semantic space (resolves which Revenue, which table, which calendar) → Catalog enforces permissions → Execution engine (SQL/Python) computes → Answer returns with provenance. This reduces the search space so the agent doesn't guess across thousands of tables each turn.

Critical Boundary: Authority ≠ Correctness

A definition used company-wide for three years may have high organizational authority yet be mathematically wrong (e.g., an erroneous distinct count). Genie Ontology solves "which definition does the company currently trust?" not "which definition is logically correct?" Ultimate reliability still depends on data quality, metric governance, and correct calculation logic.

Shift in Ontology Production: Continuous Discovery, Human Governance

The change is not "machines replace human ontology building" but "machines handle discovery; humans focus on governance." Core KPIs remain human-defined via Unity Catalog Semantics. Long-tail semantics emerge continuously from daily SQL, dashboards, and agent use. Ontology becomes a living, continuously updated system rather than a static upfront engineering project.

References

Databricks Documentation — Genie Ontology (official product docs, updated 2026-09-02)

Databricks Documentation — Unity Catalog Semantics

Databricks Documentation — Metric Views

Databricks Documentation — Pages

Databricks Documentation — AI/BI & Genie One 2026 Release Notes