Is Non-Vector RAG the Next Generation of Retrieval‑Augmented Generation?

RAG Bottlenecks

Although RAG is widely used for enterprise knowledge‑base Q&A, compliance document search, and codebase retrieval, production deployments frequently complain about relevance and accuracy problems.

Standard RAG Pipeline

Document → Chunk → Embedding → Vector DB → Similarity Search → LLM answer

Even with recent model improvements, the core issue remains that text is compressed into vectors and similarity (e.g., cosine) is used to measure relevance.

Similarity vs. Relevance

Similar but not relevant – precision loss: In domains such as law, medicine, and finance, minor wording changes can flip meaning, yet the embeddings of the two sentences overlap.

Relevant but not similar – recall loss: Truly relevant passages often use different terminology or reside deep in a document hierarchy; vector search cannot reason about structure, so such passages are often excluded from the top‑K results.

Non‑Vector RAG (PageIndex)

Core idea: Instead of a fixed retrieval flow, the query determines which structure matters. PageIndex represents each document as a tree (chapter → sub‑chapter → page → content) and lets the LLM navigate the tree to find answers.

LLM performs a binary relevance judgment at each node – “given the query, should we dive into this subtree?” – based on full‑document understanding, not on vector similarity, allowing it to cross lexical differences.

Retrieval decisions incorporate the query, conversation history, user role, and the path already taken, making the process context‑aware.

Each navigation step leaves an auditable trace: which chapters were opened, which were skipped, and which provided information, unlike opaque vector scores.

Limitations

Complexity: The non‑vector design adds topic clustering, LLM‑inferred metadata, virtual nodes, query‑driven tree construction, and cache mechanisms, which can be as intricate as vector pipelines.

Token consumption and latency: Each node’s “enter subtree?” decision requires a separate LLM call, which is serial and cannot be parallelised, increasing latency compared with the mostly offline vector retrieval stage.

Metadata quality: Tree nodes rely on LLM‑generated metadata (category, summary, key entities). Poorly structured source documents produce noisy metadata and a meaningless hierarchy.

Risk of catastrophic failure: A wrong high‑level skip decision discards an entire subtree, causing a total loss of relevant content, whereas vector retrieval failures are usually gradual.

False‑negative filtering: Non‑vector RAG may over‑filter loosely related content, leading to empty results, while vector RAG suffers from false‑positive (similar but irrelevant) hits.

When Non‑Vector RAG Is Appropriate

Best suited for document collections with clear hierarchical structure, queries that are path‑dependent, requirements for auditable retrieval paths, moderate corpus size (tens to a few hundred core documents), and scenarios where accuracy outweighs latency and non‑real‑time responses are acceptable.

Hybrid Retrieval Landscape

Current production systems typically combine vector and keyword search. Non‑vector RAG can serve as a third component in such hybrid setups, offering structural reasoning where flat similarity falls short.

Overall, non‑vector RAG represents an alternative extension of the RAG paradigm rather than a universal replacement.