Why Vector‑Based RAG Falls Short and How PageIndex’s Reasoning‑Based Retrieval Solves It

Background

Large language models (LLMs) excel at document understanding and question answering, but they are constrained by a fixed context window. As the window grows, performance degrades, making it hard for LLMs to reason over long, specialized documents such as financial reports or legal texts.

Limitations of Vector‑Based RAG

Query‑knowledge mismatch : Vector similarity assumes the most semantically similar text is also the most relevant, which is often false for domain‑specific queries.

Semantic similarity ≠ relevance : Similar passages may convey different meanings, especially in technical or legal documents.

Hard chunking : Fixed‑size text blocks break sentence and paragraph continuity, harming semantic integrity.

No chat history integration : Each query is processed independently, ignoring prior conversational context.

Cross‑reference handling : References like “see Appendix G” are missed because they lack semantic similarity to the surrounding text.

PageIndex: A Reasoning‑Based RAG Framework

Inspired by AlphaGo, Vectify AI proposes PageIndex , a non‑vector, reasoning‑driven retrieval system. It builds a hierarchical tree index of the document (using JSON) and lets the LLM navigate this structure with logical reasoning, mimicking how a human expert would browse a book.

Core Features

No vector database: Retrieval relies on document structure and LLM reasoning instead of embedding similarity.

Chunk‑free: Documents are kept in natural sections rather than arbitrary fixed‑size blocks.

Human‑like retrieval: The system simulates expert navigation through chapters and sections.

Better explainability and traceability: Each retrieved node includes page numbers and can be traced back to the source.

Retrieval Process

Read the table of contents to understand the document hierarchy.

Select the most promising chapters based on the query.

Extract relevant information from the selected sections.

Check if the information is sufficient; if not, iterate back to step 2.

When sufficient, generate a complete, well‑reasoned answer.

JSON Node Definition

Node {
  node_id: string, // Unique identifier
  name: string, // Human‑readable title
  description: string, // Optional detailed explanation
  metadata: object, // Arbitrary key‑value pairs (e.g., type, tags)
  sub_nodes: [Node] // Recursive child nodes
}

Example Command‑Line Usage

pip3 install --upgrade -r requirements.txt

python3 run_pageindex.py --pdf_path /path/to/document.pdf

# Optional arguments
--model gpt-4o-2024-11-20
--toc-check-pages 20
--max-pages-per-node 10
--max-tokens-per-node 20000
--if-add-node-id yes
--if-add-node-summary yes
--if-add-doc-description yes

Markdown Support

PageIndex can also generate a hierarchical index from Markdown files using the -md_path flag. The tool expects proper heading levels (e.g., #, ##, ###) to infer the structure.

Conclusion

By replacing static vector similarity with dynamic, reasoning‑driven navigation of a hierarchical index, PageIndex overcomes the core drawbacks of traditional RAG systems, delivering higher accuracy (98.7 % on FinanceBench) and better interpretability for long, structured documents.