OpenAI Demotes RAG: Context Graphs Become Primary for Enterprise Agents

Traditional RAG forces agents to rediscover context for every query by retrieving text chunks from vector stores. This works for simple lookups but fails when answers require cross-document, cross-system, or temporal reasoning — as demonstrated by the HERB benchmark (Salesforce AI Research, EMNLP 2025 Industry Track). HERB simulates a 530-person enterprise with 39,190 heterogeneous artifacts (documents, Slack, meetings, GitHub, URLs) and 815 answerable plus 699 unanswerable questions. Standard hybrid retrieval (dense + BM25) scores only 20.61; a GPT-4o ReAct agent reaches 32.96, with retrieval identified as the primary bottleneck. GraphRAG variants (Microsoft GraphRAG 10.31, RAPTOR 14.77, HippoRAG 2 17.21) underperform simple hybrid retrieval because they still treat the graph as a retrieval enhancer rather than a persistent knowledge layer.

OpenAI's September 21, 2026 V7 customer case study shows a different architecture: Context Graph first, RAG second . V7 continuously extracts entities (company, fund, person, document), relationships, business facts, metrics, and provenance citations into an ontology-defined Context Graph as data arrives. GPT-5.6 Luna performs the extraction and entity resolution. At query time, the agent consults the graph; only when the graph lacks needed information does it fall back to RAG over raw documents. V7 reports (self-tested) that on 1,000 private-market documents, vector-only accuracy is 72.1%, Context Graph 93.8%, and Graph + RAG 97.1%; on the hardest multi-document aggregation questions, the figures are 36.3%, 91.7%, and 95.8% respectively. These numbers come from V7's synthetic benchmark with shared ontology/question design and should be viewed as architectural illustration, not independent benchmarks.

Long-context models do not eliminate the problem. HERB's long-context experiment gave Gemini 2.5 Flash all artifacts for a single product, achieving 76.55 vs. 32.96 for agentic RAG over the full corpus — proving that complete evidence dramatically improves model performance. However, enterprises cannot fit years of emails, CRM records, data warehouses, and meeting logs into every prompt. V7's solution: keep recent dialogue in the model's active context, push older structured knowledge into the Context Graph, and traverse the graph at query time. V7 claims (without published independent verification) orders-of-magnitude cost and speed advantages over long-context reliance.

Multi-step agents amplify context errors. V7 reports agents running 50–100 steps; an early mistake (e.g., misidentifying a client at step 6, using a stale metric at step 12) cascades through downstream tool calls, judgments, and writes. V7 states (self-reported) that GPT-5.6 Sol reduced tool-call error rate from 2.7% (GPT-5.5) to 0.2% on their Context Graph benchmark, and some multi-call workflows ran up to 50% faster. OpenAI's September 10 Data Agent announcement reinforces the trend: it consumes existing semantic layers (Databricks Genie Ontology, dbt, GitHub, Snowflake Horizon, BI dashboards) for business terms, metric definitions, and data relationships. Together, these moves indicate an emerging Enterprise Context Layer beneath the model — distinct from databases and vector stores — that provides agents with persistent, queryable, provenance-tracked organizational knowledge.

RAG remains valuable for open-ended questions outside the predefined ontology, but it is retreating from the "knowledge layer" to its native role as a search layer. The new competitive frontier for enterprise AI is not model size but who can assemble the best continuously updated, traceable, agent-ready Context Graph before the agent starts working.