OpenAI's Data Agent: Why Semantic Layers Are Now Essential Infrastructure

OpenAI's Data Agent Reframes Enterprise Data Analysis

OpenAI's September 2024 Data Agent release does not merely add NL2SQL to ChatGPT; it positions ChatGPT as a unified interaction layer atop the existing enterprise data stack. The agent connects to Snowflake, BigQuery, Databricks, Google Drive, and SharePoint; consumes context from dbt, GitHub, Snowflake Horizon, and BI dashboards (Power BI, Tableau, ThoughtSpot); and can create or modify dashboards within those BI tools. This architecture signals that AI agents will not replace BI but rather depend on the governed semantic assets BI has accumulated.

Beyond NL2SQL: The Six-Layer Context Architecture

OpenAI's internal data agent, serving 3,500 users across 600 PB and 70,000 datasets, demonstrated that SQL generation alone is insufficient. Analysts spent significant time choosing among similar tables (some including anonymous users, others not), and complex joins, filter pushdowns, and null handling could silently produce wrong results—one real query exceeded 180 lines. Instead of training the model on more SQL, OpenAI built a six-layer context system:

Table Usage : schema, lineage, historical queries.

Human Annotations : domain-expert descriptions of tables and columns.

Codex Enrichment : reading the code and pipelines that generate data to understand what a table contains, how it is produced, and refresh cadence.

Institutional Knowledge : extracting product launches, incidents, internal terminology, and metric definitions from Slack, Google Docs, and Notion.

Memory : storing user-corrected filters, rules, and constraints.

Runtime Context : real-time queries to warehouse, metadata services, Airflow, and Spark when static context is stale or missing.

Offline pipelines daily consolidate Table Usage, Human Annotations, and Codex-derived knowledge into a unified representation indexed via Embeddings API for RAG retrieval. At query time, the agent retrieves the most relevant context; if insufficient, it issues live queries. This makes business context a first-class data infrastructure requiring ingestion, standardization, embedding, retrieval, and real-time validation.

Meaning Lives in Code: Semantic Fidelity Over SQL Syntax

The article emphasizes that schema tells the model a column is named revenue, and historical SQL shows how others have used it, but the definitive meaning—pipeline logic, refresh rules, filter conventions, and business assumptions—resides in the code that produces the data. An LLM can easily write SUM(revenue), but determining whether that matches the company's official revenue definition is hard: order date vs. payment date vs. finance confirmation? Refunds included? Internal accounts filtered? Trial customers counted? When CFO and sales disagree, which version is canonical? These are semantic questions, not SQL syntax questions. The production Data Agent therefore consumes Business Terms, Metric Definitions, Custom Calculations, and Data Relationships from sources like Genie Ontology, dbt, Snowflake Horizon, and BI dashboards.

BI's Interaction Layer Compressed, Semantic Layer Amplified

If BI were only a charting tool, generative AI would directly replace it. However, BI's hard-to-replicate asset is its semantic model: which tables join, how metrics are defined, how measures are calculated, which fields are exposed to business users, and which version of the numbers is blessed. Microsoft's Power BI "Prep data for AI" adds AI Data Schema (limiting fields to reduce ambiguity), Verified Answers (human-confirmed Q&A pairs), and AI Instructions (business terminology and analysis rules). Microsoft warns that without prepared data, semantic model, and user context, Copilot can give low-quality or misleading answers. Tableau's MCP integration brings Trusted Insights and Curated Semantic Models into ChatGPT, Codex, and Claude, turning AI agents into a "front door" to the Tableau Data Estate. Consequently, the semantic model's role expands: it must now ensure that different AI agents form the same understanding of Revenue, ARR, and Conversion Rate. Data modeling quality now directly determines agent trustworthiness.

Vendors Compete for the Context Layer

Databricks Genie Ontology positions itself as the unified context layer for Genie AI experiences. It separates Unity Catalog Semantics (governed, certified metric views, domains, pages) from Inferred Context (auto-extracted from metric views, dashboards, SQL queries, and Genie agents). Each inferred snippet carries an authority score based on source, usage frequency, and freshness; conflicts are resolved by ranking and only using sources accessible to the current user. This shifts data governance from "which number do people trust?" to "which context snippet should the model trust?"

Snowflake takes a different path: extracting semantics from existing BI assets. The GA of Power BI ingestion for Semantic View Autopilot (August 2024) allows enterprises to upload .pbit/.pbix files; Snowflake automatically creates Semantic Views preserving DAX measures, table relationships, and column descriptions for Cortex Analyst. Tableau workbooks (.twb, .twbx, .tds, .tdsx) are similarly parsed for tables, columns, relationships, calculated fields, parameters, filters, and custom SQL. This lets years of BI investment serve Cortex Analyst, agents, and other AI applications without rebuilding models.

The emerging stack layers: Warehouse/Lakehouse store facts; Semantic/Ontology Layer maintains metrics, relationships, and business rules; Agent handles reasoning, query, and tool use; General-purpose AI (ChatGPT) competes for the final user-facing interaction entry point. OpenAI need not own the warehouse or BI, but to be the universal interaction layer it must understand the context inside those systems.

New Evaluation Criteria for Production Data Agents

Assessing data agents moves beyond text-to-SQL accuracy. Five dimensions emerge:

Context Coverage : Can the agent locate the right tables, dashboards, code, metrics, and documents?

Semantic Fidelity : Are Revenue, ARR, Active User interpreted per the official enterprise definition?

Governance : Do user data permissions fully propagate to the agent?

Agent Loop : Upon anomalies (e.g., a query returns zero rows due to a bad join), can the agent diagnose, adjust, re-execute, and self-verify rather than stopping after one SQL generation?

Evaluation (Evals) : Continuous regression testing—OpenAI uses human-curated question/Golden SQL pairs to re-run queries after model, prompt, tool, or context updates, detecting silent quality regressions.

This mirrors the evolution of coding agents from "can it write code?" to harnesses managing context, tool use, test feedback, and long-horizon tasks. A key difference: coding agents can re-read context from code repositories, whereas enterprise business context is scattered across dbt, Power BI, Tableau, Snowflake, Databricks, data dictionaries, pipelines, documents, and analyst tribal knowledge—rules that do not automatically appear just because models improve.

Conclusion: The Next Battle Is the Business Truth Layer

ChatGPT is becoming the new BI frontend, but it does not devalue existing data infrastructure. As SQL generation and dashboard creation become commoditized agent capabilities, the metric definitions, semantic models, permissions, lineage, and business context that once lived behind reports are pushed to the core of the data system. BI's job expands from helping humans understand data to helping AI understand the enterprise. When every agent can write SQL, the competitive frontier shifts to who can provide a trustworthy, up-to-date, permission-correct, and enterprise-wide consistent "business truth layer."