Why Naive Text Chunking Breaks RAG and How to Build a Better Alternative

Problem with Naive Chunking

Technical PDFs contain tables, images, formulas and headings that have complex spatial and semantic relationships. Chunking by character count or page length breaks these relationships, producing isolated text fragments. Vector databases then return incomplete table rows, out‑of‑context figures, or disordered paragraphs, which degrades downstream large‑language‑model answer generation.

Structure‑Aware Multimodal RAG Pipeline

The pipeline restores document structure before retrieval and consists of four consecutive stages: Parse , Enrich , Ingest , and Retrieve . All stages run on local models, keeping data, embeddings and queries on‑premise.

Parse Stage

The pipeline first maps the physical layout using PP-DocLayout‑V3 (Sun et al., 2025), which detects tables, images and paragraphs and assigns precise bounding‑box coordinates. GLM‑OCR (Zheng et al., 2026) then extracts text strictly within those boxes, producing a structured JSON that preserves the original spatial hierarchy.

Enrich Stage

Non‑text elements are converted into searchable descriptions. Visual regions identified in the JSON are fed to qwen2.5‑VL:7b (Bai et al., 2025) via Ollama, which generates textual captions for each image, table or formula. The output is an enriched JSON containing original text, layout coordinates and generated captions.

Ingest Stage

The enriched JSON is vectorized using qwen3‑embedding:4b (Qwen Team, 2025). Both the text embeddings and the layout metadata are stored in Qdrant (Qdrant Team, 2024), a high‑performance vector database.

models:
  embedding: "qwen3-embedding:4b"
  llm: "qwen2.5vl:7b"
  vlm: "qwen2.5vl:7b"
  cross_encoder: "cross-encoder/ms-marco-MiniLM-L-12-v2"

Retrieve Stage

When a user query arrives, it is first converted to a dense vector and searched in Qdrant. For queries containing visual keywords (e.g., "diagram", "figure", "flowchart"), the pipeline applies a 35% score boost to image chunks (modal enhancement). In the authors' tests this moved a relevant image from rank 7 to rank 1.

is_vis = bool(set(query.lower().split()) & visual_kw)
if is_vis:
    print('   Visual query - boosting image 35%')
    for hit in results.points:
        if hit.payload.get('modality') == 'image':
            hit.score *= 1.35

For textual and tabular queries, the top‑20 candidates are re‑ranked with a cross‑encoder ( ms‑marco‑MiniLM‑L‑12‑v2 ; Nogueira & Cho, 2019). This model directly compares the query with each chunk, producing a finer relevance score that pushes the correct table to the top of the list.

Evaluation Results

Combining structure‑aware parsing, modal enhancement and cross‑encoder re‑ranking consistently places the most relevant modality (text, image, or table) at the front of the context window. For visual queries, the 35% boost raised the image chunk score from 0.837 to 1.130, improving its rank from 7 to 1. For table queries, cross‑encoder re‑ranking promoted the correct table above all plain‑text chunks.

Local Deployment Benefits

Running the entire pipeline on local hardware preserves privacy for sensitive documents, embeddings and queries, improves explainability, and simplifies debugging. Developers can inspect the intermediate JSON files from the Parse and Enrich stages to verify layout extraction and caption generation.

Implementation Overview

The workflow is orchestrated by a central run_all.py script that sequentially invokes the four phase modules ( src/phase1_parse.py … src/phase4_retrieve.py).

for phase in phases:
    print(f"
PHASE {phase}")
    if phase == 1:
        run_phase1(pdf_path)
    elif phase == 2:
        run_phase2()
    elif phase == 3:
        run_phase3()
    elif phase == 4:
        run_test_queries()

All configuration (model names, paths, visual keyword list, boost factor) lives in src/config.yaml, allowing easy adaptation to new datasets.

Conclusion

Naive PDF chunking destroys the structural cues that give meaning to tables, figures and formulas. A structure‑aware, multimodal pipeline that restores layout, generates visual descriptions, applies modality‑specific score boosts, and re‑ranks with a cross‑encoder provides richer, more accurate context for downstream LLM answer generation. Local execution further enhances reproducibility, debuggability and data privacy.

Repository

Full source code is open‑source on GitHub:

https://github.com/mohitagr18/multimodal_rag_article