5 High‑ROI Strategies to Supercharge RAG Retrieval Performance

0. What to Optimize

RAG’s main weakness lies in the retrieval stage rather than generation. Effective retrieval requires both comprehensive recall (finding all potentially relevant material) and accurate ranking (placing truly relevant material at the top). Common failure modes include semantic similarity without relevance, keyword hits without semantic proximity, fragmented chunks, unsuitable evidence formats, and stale versions.

1. Multi‑Vector Retrieval

1.1 Core Idea

Instead of simply storing more vectors, decouple the representation used for retrieval from the original content used for answer synthesis:

Recall phase: use vectors optimized for similarity search (e.g., summaries, question‑style descriptions, natural‑language table captions, image alt‑text).

Generation phase: feed the original material (full text, complete tables, original images) to the LLM so that details are not lost.

This extends RAG beyond pure text to semi‑structured or multimodal content such as tables and images: retrieve with summaries, answer with the original source.

1.2 When It Pays Off

Multi‑vector retrieval consistently benefits three data categories:

Semi‑structured documents: tables mixed with paragraphs (financial reports, audit reports, policy documents).

Multimodal documents: images, charts, scanned PDFs (manuals, bid documents, reports).

Long documents: a single topic spread across many chapters, where a single chunk embedding may miss relevant context.

1.3 Engineering Implementation

Split by element type: partition documents into text blocks, tables, images, etc. Tools like Unstructured can first extract image regions, then detect table boundaries and headings, finally aggregate surrounding text.

Generate searchable text for each element:

Text blocks – short summaries, keyword‑style descriptions, possible question sets.

Tables – natural‑language summary of what the table conveys (used for retrieval).

Images – multimodal models convert images to textual captions for indexing.

Keep the original source in the docstore: retrieval returns the summary, but the LLM receives the full original content (text, table, or image reference) for answer synthesis.

1.4 Common Pitfalls

Summaries that are too close to the original text add noise without improving recall; make them concise, structured, and include entities, metrics, and time ranges.

Generating too many vectors per element inflates storage and latency; aim for sufficient coverage, not exhaustive duplication.

Unstable mapping between summary IDs and original IDs leads to mismatched evidence; design stable, reproducible primary keys from day one.

2. Manual Splitting & Labeling

2.1 Why Human Intervention Is Needed

Fully automatic ingestion often fails because:

Uniform splitting rules perform unevenly across document types.

Critical structural information resides in layout, headings, or table schemas rather than plain text.

Business‑critical metadata (version, scope, region, product line) is not directly extractable from the body.

Manual splitting and labeling solidify the structure and semantics required for effective retrieval, allowing downstream vectorization and ranking to work properly.

2.2 Three Splitting Rules

Split by semantic boundaries, not length: use chapters, sections, clauses, definitions, FAQs as natural cut points; avoid arbitrary token‑based cuts that break definitions.

Granularity should serve evidence citation: keep the smallest unit that can be referenced, such as clause numbers, table titles + whole table, subsection titles + body, or whole definition paragraphs.

Preserve hierarchy: store hierarchy levels (document → chapter → section → paragraph/table) rather than a flat list of chunks, enabling upward context expansion later.

2.3 Prioritize Filterable & Routable Tags

Instead of exhaustive ontologies, start with high‑value tags:

Document type (policy, product manual, contract template, meeting minutes, financial report).

Business scope (region, product line, customer type, applicable system).

Temporal attributes (effective date, version, deprecation status).

Reliability (source system, approval status, official release).

Access control (department, role, confidentiality level).

2.4 Common Tagging Pitfalls

Inconsistent tag vocabularies – enforce a whitelist of allowed keys and enumerate or regex‑validate values.

Using “topic” as a tag – topics are unstable and overlap with embeddings; reserve tags for hard constraints and business boundaries.

3. Scalar Enhancement

Scalar fields (time, version, source weight, permission, quality score, business line) provide controllable filters and re‑ranking logic that complement pure vector similarity.

3.1 Problems Addressed

Same question yields different answers over time – old versions cause failures.

Same concept varies across regions or product lines – vector similarity cannot distinguish.

Noisy documents inflate similarity scores.

Need for explainability and auditability of evidence selection.

3.2 Two Common Approaches

Approach A – Filter then retrieve: apply metadata constraints (effective date ≤ query time, matching version, product line inclusion, permission check) before vector recall.

Approach B – Re‑score after retrieval: obtain top‑K vectors, then adjust scores using scalar rules (newer docs get a boost, official sources get higher weight, citation count or manual verification adds points).

Combine the scalar and vector scores into a final ranking.

3.3 Key Practices

Make scalar fields maintainable – prefer automatic extraction, then semi‑automatic, finally manual.

Use conservative default values; missing fields should not add points.

Log every filtering and re‑scoring decision in production for easier troubleshooting.

4. Context Augmentation

When chunks are split, essential surrounding information may be lost, leading to “orphan sentence” errors. Context augmentation adds the necessary background to each retrievable unit.

4.1 Scenarios Where Context Is Missing

Regulations referencing earlier definitions.

Financial reports using abbreviations defined only once.

Table field meanings explained in a preceding caption.

Meeting minutes where “agree/disagree” needs the referenced participant.

4.2 Three Implementation Patterns

Pre‑embedding lightweight context: prepend title path, section name, and document name to the chunk before embedding.

Parent/Window expansion: after recalling a small chunk, fetch its parent node (section/chapter) or a surrounding window of chunks.

Structured index / tree retrieval: build a hierarchical index (e.g., PageIndex) that first locates the relevant node in the tree and then drills down to the exact paragraph, eliminating the need for a flat vector DB.

4.3 Common Pitfalls

Appending too much context dilutes the embedding signal; include only the most discriminative pieces (title path, short definition, field explanation).

Unbounded window expansion can feed an entire chapter to the model, increasing cost and noise; set a clear limit and prefer same‑section expansion over whole‑document.

5. Combining Dense and Sparse Vectors

Dense embeddings excel at semantic similarity, while sparse retrieval (BM25) excels at exact keyword matching. Integrating both mitigates each method’s weaknesses.

5.1 Integration Strategies

Approach A – Parallel recall + merge + re‑rank: retrieve top‑K results from BM25 and from dense vectors, merge, deduplicate, then apply a unified re‑ranker or simple rule‑based ordering.

Approach B – Two‑stage (BM25 first, dense second): use BM25 to narrow the candidate set, then apply dense similarity and re‑ranking on the reduced set, saving vector search cost for large corpora.

5.2 Common Pitfalls

Relying on BM25 as the primary engine loses recall for paraphrased queries.

Naïvely adding BM25 and dense scores without normalization leads to instability; normalize scores before merging or let a learned re‑ranker decide.

Poor Chinese tokenization degrades BM25; enrich the dictionary with product names, abbreviations, and field identifiers.

6. Summary

The five strategies can be adopted incrementally:

Start with manual splitting and labeling to clean structure, version, permission, and scope.

Integrate BM25 to guarantee keyword hits.

Apply context augmentation to fix fragmented evidence.

Introduce scalar enhancement for controllable, explainable ranking.

Deploy multi‑vector retrieval to handle tables, images, and long documents.

In practice, production systems layer these techniques: BM25 provides precise matches, dense vectors supply semantic recall, scalar logic enforces business constraints, context augmentation ensures evidence coherence, and multi‑vector retrieval bridges across modalities.