RAG's Hidden Flaw: Microsoft Shows Knowledge Graphs Raise Accuracy from 26% to 93.5%
Microsoft's experiments on SEC filings reveal traditional RAG retrieves 72% of evidence but achieves only 26.1% answer accuracy due to incomplete fact retrieval; integrating ontology-driven knowledge graphs (Entity Graph + Topic Graph) raises accuracy to 93.5% and cuts token usage by 64%, though ontology design requires iterative alignment with business queries.
Microsoft recently published a series of experiments that expose a critical limitation of traditional Retrieval-Augmented Generation (RAG) in enterprise settings: even when the retriever finds most relevant passages, the final answer can still be wrong because the model lacks a complete set of facts.
1. The Evidence Recall Gap
On six categories of structured queries over SEC filings — counting & calculation, source tracing, pattern & anomaly, shared relationships, difference & missing, and negative statements — traditional RAG achieved an average evidence recall of 72.0% but a complete evidence recall of only 26.8% . Complete evidence recall requires that all facts needed for a single query are retrieved; missing even one fact (e.g., one product in an intersection query) breaks the answer. Consequently, the final answer accuracy was only 26.1% .
For direct fact lookup (contract clauses, timelines, org charts) both RAG and knowledge graphs scored ~96–98% accuracy, but RAG consumed 3.1× more tokens. The gap appears when answers depend on complete relationships across multiple entities .
2. Ontology-Driven Knowledge Graph Construction
Microsoft's Foundry IQ introduces Knowledge Graph Entities . An Ontology defines entity types (Organization, Product, Business Segment), relationship types (offers, belongs_to), and a schema constraining their structure. LLMs then extract entities and relations from documents (financial reports, contracts, etc.) according to that schema, populating an Entity Graph that stores structured facts with provenance links back to source passages.
This shifts the burden from runtime retrieval to offline structuring: the agent queries pre-organized relationships instead of re-deriving them from text each time. The system also supports GQL (Graph Query Language) for programmatic access.
3. Ontology Iteration: Diminishing Returns on Fact Coverage
Microsoft ran three ontology versions on the same SEC corpus, each followed by five extraction runs (averaged):
Initial (auto-generated): 38 entity types, 78 relationship types, 5,474.6 deduplicated facts.
First optimization (aligned to common SEC queries): +5 entity types, +22 relationship types, facts jump to 7,235.2 — 81% of total fact gain .
Final version (extended financial expression): +6 entity types, +34 relationship types, facts reach 7,656.2 — only marginal further increase.
Relationship types grew 72% overall, but fact extraction plateaued after the first targeted optimization. The lesson: ontology design must be driven by actual query patterns , not just exhaustive schema expansion.
4. Entity Graph + Topic Graph Collaboration
For complex cross-period analysis (e.g., “What actions did Microsoft take when AI demand exceeded infrastructure capacity across July 2024 and January 2025 earnings calls?”), Foundry IQ combines three retrieval modes:
RAG — direct passage retrieval.
Entity Graph — structured relationship queries.
Topic Graph — built from document embeddings, organizes topics/subtopics into a sitemap to diversify retrieval across themes and time.
On a 24-fact cross-period task:
RAG alone: 64.2% evidence recall, 15.4 facts, 212,718 tokens.
RAG + Entity Graph: 81.2% recall, 19.5 facts, 147,781 tokens.
RAG + Entity Graph + Topic Graph: 92.0% recall, 22.1 facts, 102,288 tokens (52% token reduction).
Note: 92.0% is evidence recall on this task, not directly comparable to the 93.5% answer accuracy on the structured-query benchmark.
5. Operational Considerations
Permissions: Foundry IQ enforces source-level ACLs (SharePoint/Entra, ADLS Gen2, Azure Blob RBAC) and preserves provenance in the graph. Cost: Token figures reflect query-time only; preprocessing, LLM extraction, ontology maintenance, and graph updates add ongoing overhead. Maintenance: Ontology must evolve with business entities and query needs — missing relationships in the schema mean missing facts in the graph, even if documented.
Source: Microsoft Foundry Blog, Nov 17, 2024 — “Foundry IQ: Introducing knowledge graphs, engineered for agents” and “Knowledge graphs in Foundry IQ improve evidence recall with fewer tokens”.
Signed-in readers can open the original source through BestHub's protected redirect.
This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactand we will review it promptly.
How this landed with the community
Was this worth your time?
0 Comments
Thoughtful readers leave field notes, pushback, and hard-won operational detail here.
