Why Top AI Coding Agents Dropped Vector Databases for Grep-Based Retrieval

In May 2025, Anthropic removed vector search from Claude Code, replacing embeddings, a local vector database, and chunking heuristics with a single grep command. According to Claude Code author Boris Cherny on the Latent Space podcast, the result outperformed everything. By a lot, and this was surprising .

How Vector Retrieval Failed on Code

The SWE-bench baseline (October 2023) used a simple RAG pipeline: chunk codebase, embed, retrieve top-k, generate patch. It scored only 1.96% . The first agent system, SWE-agent, swapped retrieval for tools ( open_file, scroll_down, edit_lines) and jumped to 12.47% . By 2026, the SWE-bench Verified leaderboard is dominated by agentic systems; none of the top entries rely on vector retrieval.

The article identifies five reasons vector retrieval fails on code:

Semantic similarity ≠ relevance. The most similar embedding is a poor proxy for "which function breaks if I change this line?" Code has explicit structural relationships (imports, type definitions, call graphs) that flat embeddings flatten.

Identifiers are search. When you ask where processPayment is defined, you need exact match. Vector search yields false positives ( handlePayment) and false negatives (the real definition pushed down by a look-alike comment).

The index is always stale. Code drifts on every commit; continuous re-indexing is expensive and never catches up — the same problem GraphRAG faced before LazyGraphRAG.

The index is a liability. A private codebase's vector index is a copy of that code sitting on separate infrastructure, often with weaker access controls than the source repo.

Single-shot retrieval is brittle. Top-k gets one chance; if the first miss fails to hit the right file, the model confidently generates wrong code.

Amazon's AAAI 2026 paper ( Keyword Search Is All You Need ) generalizes this beyond code across six datasets (FinanceBench, BlockchainSolana, Llama2Paper, HistoryOfAlexnet, etc.). On FinanceBench, agentic keyword retrieval beat traditional RAG by 6 percentage points (30.40% vs 24.24%) . The failure mode of chunk+embed is universal, not code-specific.

Four Reasons Anthropic Swapped RAG for grep

Accuracy — the biggest surprise. The team expected agentic retrieval to be worse and accepted a quality trade-off for operational simplicity. Instead it won. An LLM driving iterative grep can refine queries, read neighboring files, follow imports, and self-correct — impossible with a single embedding lookup.

Freshness. The agent reads the filesystem directly, reflecting the repo's current state with zero index lag. Change a file, wait 100 ms, ask Claude Code — it reads the new bytes. Vector indexes wait for the next re-embedding cycle.

Security & privacy. Cherny: "RAG has a whole indexing step… the index has to live somewhere… that's a huge liability for companies." Enterprises especially don't want a separate embedded copy of proprietary code on someone else's infrastructure.

Reliability. Fewer components = fewer failures. Grep-based retrieval has no drifting embedding model, no vector database to go down, no lagging rebuild pipeline, no chunking strategy to tune. ripgrep, find, cat just work.

Just-in-Time Context Loading

Anthropic's September 2025 engineering blog Effective context engineering for AI agents names this pattern just-in-time (JIT) context loading . It draws a sharp line against traditional RAG:

Pre-inference retrieval (RAG): Pre-embed everything, store vectors, query-time fetch top-k, stuff into prompt. Must predict and index everything the model might need upfront.

JIT loading: Agent maintains lightweight identifiers (file paths, stored queries, URLs) and dynamically loads referenced content at runtime via tools. Nothing pre-loaded; fetch only what the agent judges relevant, when it judges it relevant.

This changes the shape of the context window. In pre-inference RAG, tokens are spent on chunks you guess are relevant; in JIT, tokens are spent only on chunks the agent judges relevant. Anthropic summarizes it as "find the smallest, highest-signal token set that maximizes expected outcome." Naive agent loops use more tokens than vector RAG, but JIT loading combined with sub-agent context isolation can use fewer tokens by completely avoiding low-signal chunks. An internal figure: Claude Code's lazy tool loading (loading tool definitions only when needed) reduced context consumption by ~95%.

Agent as Retriever: A Deliberately Boring Toolset

In Claude Code, retrieval is exposed as a small set of deliberately boring tools:

Glob : file-path pattern matching, near-zero token cost.

Grep : regex content search via ripgrep (macOS/Linux native builds switched to built-in ugrep + bfs from April 2026).

Read : load full or partial file content into context.

Bash : fallback shell for long-tail ops ( tail, head, jq, git log, find with predicates).

Explore sub-agent : a read-only independent agent (default Haiku 4.5) with its own context window for parallel codebase exploration.

A reverse-engineering study of Claude Code's TypeScript source counted 54 built-in tools (19 unconditional, 35 feature-gated), yet only 1.6% is AI decision logic ; the remaining 98.4% is ops infrastructure, context management, permissions, tool dispatch, compression. The decision layer is tiny; the retrieval and context management layer is huge.

The control loop shape:

plan → glob/grep → read candidates → refine query
     → repeat (or spawn subagent) → compact → answer

This mirrors agentic RAG (Self-RAG, CRAG, A-RAG) with one key difference: no pre-built index between the agent and the bytes . The retriever is the shell tool the agent chooses to call.

Because JIT loading eventually fills the context window, Claude Code runs a five-stage compression pipeline near the 200k token limit:

Budget reduction : drop least relevant content first.

Snip : remove redundant tool-call outputs.

Microcompact : summarize over-long individual messages.

Context collapse : fold earlier turns into shorter retrospectives.

Auto-compact : final summarization when nothing else fits.

Three Benchmarks That Prove the Point

Amazon AAAI 2026: Keyword Search Is All You Need

Same LLM (Claude 3 Sonnet, 200k context, temp 0.001), same six datasets, same eval framework. Only difference: retriever. One side uses Amazon Bedrock Knowledge Base with Titan Text Embeddings V2; the other uses a ReAct agent calling pdfmetadata, rga, pdfgrep.

Faithfulness: agent 0.81 vs RAG 0.86 → 94.5% of RAG.

Context Recall: agent 0.68 vs RAG 0.77 → 88.0% .

Answer Correctness: agent 0.59 vs RAG 0.65 → 91.5% .

Conclusion: vector databases are not necessary for high-quality retrieval; agentic keyword search is a viable alternative for many applications.

Search-R1: RL-Trained Retrieval Policy

Search-R1 goes further: train the retrieval strategy itself with RL. Give an R1-style reasoning model the ability to emit <search>query</search> mid-reasoning; use outcome-based RL (veRL + RAGEN) to teach when to search, what to search, when to stop . Retrieval tokens are masked during training for stability.

On seven QA datasets (NQ, TriviaQA, PopQA, HotpotQA, 2WikiMultiHopQA, Musique, Bamboogle) with Qwen2.5-7B:

Search-R1 avg EM: 0.431 , RAG baseline 0.304.

Relative gain: 24% (3B model: 20%).

Reference points: SFT 0.207, no-retrieval R1 0.276, rejection sampling 0.348.

Architectural significance: once retrieval is a tool call, it becomes a learnable policy . You can use the same RL machinery that produces reasoning models to train better retrievers — impossible with frozen embedding models.

Anthropic's Multi-Agent Research System

Orchestrator-worker architecture: a lead (Claude Opus 4) decomposes queries, generates plans, dispatches 3–5 sub-agents in parallel; each sub-agent (Claude Sonnet 4) runs its own agentic retrieval loop, calling 3+ tools per turn; sub-agents return only distilled conclusions, full tool traces isolated inside sub-agent contexts.

Result: this multi-agent system beats a single Claude Opus 4 by 90.2% on internal research evals , cutting complex-query research time up to 90%, at ~15× token cost. Anthropic notes token usage itself explains ~80% of performance variance — when task value justifies the spend, throwing more tokens at agentic retrieval yields near-linear quality gains. Caveat: multi-agent helps breadth-first tasks (research, "find all X across sources") but hurts depth-first serial tasks like coding.

Five Architectural Variants of Agentic Retrieval (2026)

Pure agentic (Claude Code, Devin) : no persistent index; only Glob, Grep, Read, Bash, Explore sub-agent. Bet: an LLM driving ripgrep in a loop beats any frozen embedding model on a codebase that changes every commit.

Hybrid lexical + semantic (Cursor, Sourcegraph Amp) : Cursor docs list both modes — exact symbols via Instant Grep, conceptual queries via semantic search; agent picks by query shape. Cursor cites internal research: semantic + grep = +12.5% precision. Amp layers the same agent atop Sourcegraph's long-maintained code graph. Hybrid is where most enterprise tools converge.

Structural / AST-aware (Cline, Probe, ast-grep) : pure grep is lexical, pure embeddings semantic; third path is structural. ast-grep, Probe use tree-sitter to parse code, letting agents search by syntactic patterns not strings. Example: "find every fetch().then(...) and rewrite to await " — queries grep cannot express. Cline's open-source implementation is a clean three-layer stack: (1) ripgrep content search (with output caps), (2) fzf fuzzy file/dir search (custom scoring), (3) tree-sitter AST extraction for multi-language definition discovery. Agent orchestrates all three in a plan-and-act loop, weighting the currently open file higher. Cline reports this keeps per-turn retrieval token usage at ~17.5% while preserving structural awareness.

Specialized retrieval models (Windsurf SWE-grep, Chroma Context-1) : train a small model dedicated to retrieval. Windsurf's Wave 13 (early 2026) released SWE-grep and SWE-grep-mini: 8 parallel tool calls per turn, 4 turns, 10× faster than generic agentic retrieval. Chroma's Context-1 is a 20B agentic retrieval model; ~10× faster inference, ~25× lower cost on same multi-hop tasks.

RL-trained retrieval policies (Search-R1, CoSearch, Agentic-RAG-R1) : agents learn when to retrieve, what to retrieve via RL, beating prompted agents by double digits. This is the cleanest theoretical justification for agent-as-retriever long-term: it's a learnable system , not a fixed pipeline.

All five share one architectural premise: the agent owns retrieval . Differences lie only in what backs the tools and how those tools are built.

Tool Design Principles & MCP

Agent-as-retriever lives or dies by tool design. Anthropic's context engineering blog states principles bluntly:

Tools must be self-contained, error-resistant, and have crystal-clear purpose .

Input parameters must be descriptive, unambiguous, and play to the model's native strengths .

Avoid bloated overlapping tool sets; each tool does one clear thing.

Ultimate test: if a human engineer can't clearly say which tool fits a scenario, don't expect an AI agent to do better .

That's why Claude Code's tool surface is tiny. Glob does one thing, Grep does one thing, Read does one thing, Bash handles the rest (with explicit permission gates). The model never chooses between find_file_by_name, search_file_by_path, locate_file — one tool per problem shape.

Model Context Protocol (MCP) turns this pattern from a Claude Code feature into an ecosystem default. Introduced by Anthropic in November 2024, MCP is a JSON-RPC 2.0 protocol letting any LLM-powered host (Claude Code, Cursor, VS Code, Claude Desktop) connect to arbitrary MCP servers — programs exposing tools, resources, prompts to the host.

The official MCP filesystem server is a clean production example: it exposes a carefully trimmed toolset — read_file, write_file, list_directory, search_files, get_file_info — under an explicit allow-listed directory tree. The agent decides which to call, not the protocol.

Implication: any MCP-aware host becomes an agent-as-retriever system for anything with a filesystem shape . Sentry exposes incidents, Postgres exposes tables, filesystem server exposes repos — the agent treats them uniformly: discover, search, read, refine. Once retrieval is a tool call, every data source becomes a candidate retriever — no one needs to build a vector index for it.

When Not to Rush Throwing Away Your Vector DB

The pattern isn't a free lunch. Serious critiques:

Token Cost

Anthropic's own data: multi-agent research uses 15× tokens vs chat. Milvus published "Why I'm Against Claude Code's Grep-Only Retrieval" arguing iterative grep loops cost far more per query than pre-computed lookup. Industry estimate: 5–30× tokens per task vs chat; complex agent loops $0.02–$0.10 per query vs pennies for vanilla RAG. Prompt caching and lazy tool loading recover a large chunk, but not all.

Latency

5–10 tool calls per query = seconds, not milliseconds. Fine for interactive coding; fails sub-second user chat. SWE-grep and Context-1 exist precisely to compress latency.

Massive Corpora

grep on 10M-file monorepos isn't free. ripgrep and parallel traversal help; Explore sub-agents fan out; but at petabyte scale pre-computed indexes still win. Hybrid answer: run agentic retrieval inside a smaller, agent-chosen slice.

True Semantic Queries

"What does this codebase say about retry policies?" Harder for grep than embeddings when answers scatter across files that never use the word "retry" ( backoff, requeue, circuit_breaker). Agent-as-retriever answer: issue multiple queries and synthesize; Probe and AST tools answer via structural understanding; Cursor's hybrid path keeps a light semantic layer for synonyms.

Tightly Coupled Tasks

Anthropic's own caveat: multi-agent helps breadth-first problems, hurts depth-first serial tasks like coding. Work shape matters.

Determinism & Caching

Vector lookup is deterministic, cheap, cacheable; agent loops are not. Production teams converging on RAGAS, BenchmarkQED, SWE-bench Verified, but eval, regression testing, and SLA enforcement are harder than for static retrievers.

2026 Default: Tools First, Indexes Later

A decision matrix for production teams in 2026:

Private repo code → agent-as-retriever (pure or hybrid). Claude Code, Devin, Cursor converge here.

Large enterprise monorepo + cross-service queries → hybrid (Cursor, Sourcegraph Amp) or structural (Probe).

Continuously changing corpora (logs, dashboards, CRM, tickets) → agent-as-retriever via MCP server; freshness over peak recall.

Stable knowledge bases (product docs, FAQs, glossaries) → vector RAG with reranker; don't over-engineer.

Long, layout-heavy PDFs (financial reports, papers, contracts) → ColPali/ColQwen2 visual retrieval, or agent-as-retriever with pdfgrep; both work.

Whole-corpus thematic / global questions → LazyGraphRAG. Agent-as-retriever struggles when no single query can hit the answer .

Multi-hop reasoning + dynamic retrieval → Search-R1 style RL retrieval policies, or agentic RAG frameworks.

Latency-sensitive chat → specialized retrieval models (SWE-grep, Context-1), or hybrid with vector fallback.

Breadth-first research → multi-agent + agentic retrieval; 90.2% gain, 15× tokens; choose when task value > token cost.

Strongly serial tasks (end-to-end feature) → single agent + strong context engineering; don't pay multi-agent overhead.

Strict data residency / compliance → agent-as-retriever structurally easier to approve: no external index, no embeddings leaving host.

For three years the mainstream RAG assumption was: retrieval is an upstream system for the LLM — chunk, embed, top-k, prompt. Agent-as-retriever and its formalization as JIT loading flip it: retrieval is the LLM's behavior, expressed via tool calls . The agent decides what to find, when to re-find, when to stop, how to combine findings. The retriever is whatever shell command, MCP server, AST query, or RL-trained policy best fits the moment.

Evidence isn't speculative. Claude Code shipped it and saw it beat RAG; Anthropic's own multi-agent system uses the same pattern for +90.2%; Cursor responded by hiring the people who built it; Amazon hit 94.5% of vector RAG faithfulness with zero vector DB; Search-R1 used RL to train the retrieval policy and beat RAG by 24%. The pattern is being replicated, benchmarked, productized, standardized via MCP, and end-to-end trained.

None of this means vector search is dead. It means vector search is no longer the default. The 2026 default is: give the agent tools, design the tools well, let it retrieve just-in-time ; only add vectors back on workloads that genuinely need semantic generalization, massive stable corpora, or sub-second chat. Teams that see this clearly and start running leaner stacks will benefit first; those that don't will spend 2026 maintaining vector indexes for problems that never needed them.