Beyond Lakehouse: How Databricks Is Building an Agent Operating System

Over the past decade Databricks has guided enterprise data architecture from separate data lakes and warehouses toward a unified Lakehouse. Delta Lake adds ACID transactions, schema enforcement, scalable metadata, and unified batch‑stream processing, while the Lakehouse aims to let data engineering, analytics, and machine learning share a single open data foundation.

In the Agent era the bottleneck moves from merely unifying data to providing the enterprise context that agents need to understand concepts such as “revenue”, “customer”, or “order risk”. An agent must grasp metric definitions, entity relationships, real‑time state, permission boundaries, and produce an auditable evidence chain for any action it takes.

Databricks refers to this emerging “Agent operating system” not as an official product name but as an industry judgment: a set of responsibilities that allocate context, state, tools, identity, permissions, and execution boundaries to enterprise agents.

Four‑layer Agent data stack

Layer 1 – Facts & State : Delta Lake and the Lakehouse continue to store massive historical data and support batch‑stream workloads. In January 2026 Databricks released Lakebase, a managed PostgreSQL service for low‑latency transactional use cases, online features, and persistent agent session state. Lakebase changes are streamed to Unity Catalog Delta tables via Lakehouse Sync (Change Data Feed), which remains in Public Preview.

Layer 2 – Business Semantics : Unity Catalog Business Semantics centers on Metric Views, consolidating KPI definitions, dimensions, relationships, and calculation logic for both BI tools and agents. Genie Ontology automatically extracts knowledge fragments (metric definitions, authoritative sources, business rules) from tables, queries, dashboards, and documentation, assigning authority scores and filtering by permissions; this capability is also in Public Preview.

Layer 3 – Runtime Governance : Unity Catalog now governs Agent Services, MCP Services, model services, functions, and external connections as first‑class, governable objects. Unity AI Gateway routes model and MCP requests, enforcing access control, rate limits, budgets, guardrails, and audit logging. Both the unified governance objects and the AI Gateway are currently in Beta.

Layer 4 – Agent Construction & Execution : Agent Bricks provides the entry point for building, optimizing, and orchestrating agents. The broader Databricks Agent stack lets developers use multiple models and third‑party frameworks, integrates with MLflow for tracking and evaluation, supports Databricks Apps deployment, and offers Managed Agent Memory (Beta) for cross‑session long‑term memory. Individually these pieces are not a full OS, but together they assume OS‑like duties: resource management, state maintenance, identity enforcement, and interaction constraints.

Comparison with peers

Palantir’s Ontology tightly couples data, logic, actions, and security, providing a more mature action‑oriented model than Databricks’ preview‑stage capabilities.

Snowflake’s Cortex Agents (GA November 2025) enable Semantic Views, search, code execution, custom tools, and remote MCP connections, positioning Snowflake as a fully governed data‑cloud agent platform.

Microsoft Fabric combines Power BI semantic models, OneLake, Copilot Studio, and an Operations Agent (GA) that can send Teams notifications, run pipelines, notebooks, user‑defined functions, or Power Automate flows. Its Ontology integration is still in Preview, and the Operations Agent’s actions are limited by creator‑delegated identities and permissions.

Databricks’ advantage lies in its open Lakehouse, data‑engineering pipelines, and AI development ecosystem, but its ontology, governance, and memory features remain in preview or beta, and it has not yet replaced existing transaction systems.

Strategic implications

The competitive focus is shifting from model performance to who controls the enterprise context that agents consume and the permissions that govern their actions. Databricks aims to become the indispensable control plane for agents—providing the data, semantics, identity, tool access, and audit mechanisms—while the maturity of its preview features will determine whether it can truly bridge the gap from “understanding data” to “managing actions”.

References: Databricks (2026) Agent Bricks presentation; Unity Catalog Business Semantics; Genie Ontology (Public Preview); Unity AI Gateway (Beta); Lakebase GA release notes; Palantir Ontology system; Snowflake Cortex Agents GA; Microsoft Fabric Ontology integration.