How Anthropic Achieves 95% Accuracy in 95% of Data Agent Scenarios

Anthropic published an internal case study on using Claude for self‑service data analytics. The authors argue that the real challenge is not SQL generation but reliably routing vague business questions to a governed, verifiable answer.

Why Data Analysis Differs from Coding

The difficulty lies in determining which tables, metrics, and business definitions to use, and how to prove the answer is trustworthy. Unlike coding agents, which have clear compile‑time checks, data analysis must validate that the chosen data source, filter, and time window are correct, because a numerically precise result can still be wrong.

Three Primary Failure Modes

Anthropic identifies three error categories that match common enterprise pain points:

Conceptual ambiguity : questions like “active users” can refer to logins, clicks, payments, or other definitions, and the look‑back window may vary.

Stale data or metrics : table schemas, metric definitions, and organizational structures evolve, so documentation must be continuously maintained.

Correct answer not found : even when the answer exists in the corpus, the agent may fail to retrieve it because the structure of the information is insufficient.

Anthropic’s experiments show that simply feeding the agent large amounts of historical SQL improves accuracy by less than one point, indicating that “more context” is not the solution.

Agentic Analytics Stack

The proposed stack consists of three layers:

Data foundations : canonical datasets, standardized data models, CI validation, freshness checks, and metadata management. This layer emphasizes that data governance becomes even more critical when agents, not humans, query the warehouse.

Sources of truth : a semantic layer, lineage graph, transformation graph, curated query corpus, and business context that tell the agent which source to trust and when to involve a human.

Skills : procedural knowledge that encodes analyst workflows (e.g., how to narrow scope, when to consult the semantic layer, how to perform adversarial review). Adding Skills lifts Claude’s accuracy from ≤21 % to >95 % overall, with some domains approaching 99 %.

Governance and Continuous Maintenance

Because data models and documentation change, the “source of truth” must be versioned and linked to the same engineering process. Anthropic reports that roughly 90 % of data‑model PRs also include a corresponding Skill change, enabling automated review hooks to catch mismatches.

Evaluation Methodology

Anthropic stresses a three‑pronged evaluation:

Offline eval that aims for near‑100 % pass rates before any production rollout.

Ablation studies that isolate the impact of adding documentation, Skills, or reviewer sub‑agents.

Online monitoring that records provenance, freshness, owner, and confidence for each answer, ensuring users can judge trustworthiness.

Product Implications for Enterprise AI

The authors conclude that a useful AI analytics assistant must do more than write SQL faster; it must route questions to a unified metric definition, expose verification metadata, and reduce false confidence. Data teams will shift from ad‑hoc query work to maintaining a trustworthy analytics infrastructure that includes governance, evaluation, and feedback loops.

Key Takeaway

AI‑driven self‑service analytics succeeds not by making the model more autonomous, but by compressing deep data‑governance knowledge, procedural Skills, and verification mechanisms into a stable Agent that consistently delivers correct, fresh, and auditable answers.