RAG with Multimodal Inputs vs LLM + Toolchains: Handling Non‑Text Data

LLM Fundamentals

All Tokens

LLMs operate on tokens, not characters or words. A BPE tokenizer may split "tokenization" into ["token", "ization"], each token mapped to a high‑dimensional vector.

Transformer Core: Self‑Attention

Self‑attention computes a relevance weight between every pair of tokens, enabling the model to capture long‑range dependencies such as subject‑verb relationships across a sentence.

Autoregressive Generation

GPT‑style models generate text token by token: each step predicts the next token distribution, samples a token, appends it to the input, and repeats. Both input and output are token sequences.

LLM Capability Boundaries

Current LLMs excel at semantic understanding, logical reasoning, text generation, code generation, and structured output, but they cannot directly ingest non‑text modalities because those modalities lack a token representation.

LLM + Toolchain Approach

Toolchain Concept

Convert any source data into plain text, then feed the text to an LLM.

Advantages

Cost‑controlled : Text extraction tools are cheap; OCR APIs cost far less than sending images to multimodal models. At million‑document scale the cost gap spans orders of magnitude.

Fast : Deterministic parsers run in milliseconds, avoiding model inference latency.

Highly controllable : Parsed output can be cleaned, filtered, and formatted before LLM consumption, making error sources easy to locate.

Long‑document friendly : Documents can be chunked arbitrarily and indexed with vector search, whereas multimodal models face context‑window limits.

Simple deployment : Open‑source parsers run locally without large‑model infrastructure.

Drawbacks

Information loss at the parsing layer :

Image‑text relationships disappear when a chart is replaced by [图片] or garbled OCR.

Typographic hierarchy (heading size vs body) is lost.

Complex tables with merged cells or multi‑level headers are often mis‑aligned.

Scanned documents rely entirely on OCR; low image quality, handwriting, or stamps are rarely recognized.

Poor format robustness : Different PDF generators produce varied internal structures; a parser that works on one may fail on another.

Maintenance overhead : Each file type requires a dedicated tool; version upgrades or format changes increase the maintenance burden.

Chart semantics missing : OCR may only read axis numbers, losing trend information that only a visual model can capture.

Multimodal Model Fundamentals

Processing Different Modalities

Multimodal models add non‑text input channels and align their outputs with text tokens so the LLM backbone can reason over a unified sequence.

Vision Encoder: ViT and CLIP

ViT splits an image into fixed‑size patches (e.g., 16×16), flattens each patch, and maps it to a vector, creating a "visual token" sequence.

CLIP aligns visual and textual embeddings via contrastive learning on image‑caption pairs, ensuring that matching pairs are close in the shared space.

Audio Encoder

Audio is transformed into a mel‑spectrogram (a 2‑D matrix) and processed similarly to an image, e.g., by Whisper’s convolution‑Transformer pipeline.

Cascade approach : ASR first converts audio to text, then the text is fed to an LLM (used by LLM + toolchain setups).

End‑to‑end approach : Directly encode audio into vectors and feed them to a multimodal LLM (e.g., GPT‑4o, Gemini).

Cross‑Modal Alignment: Projection Layer

Visual and audio vectors have different dimensions from LLM text embeddings; a projection layer maps them into the same semantic space. Models like LLaVA train this layer on massive image‑text pairs.

Training Stages

Alignment training : Freeze the LLM backbone, train only the projection layer on large image‑text datasets.

Instruction fine‑tuning : Unfreeze part or all parameters and fine‑tune on high‑quality multimodal instruction data (image‑text Q&A, document understanding).

Comparing the Two Solutions

Fundamental Difference

LLM + toolchain first converts files to text, then the LLM interprets the text. Multimodal models ingest the original modality directly, avoiding the conversion step but incurring higher compute cost and less transparency.

Cost and Hallucination

Images consume many more tokens than text; a typical image may cost hundreds of tokens, making processing several times more expensive than pure text.

Multimodal hallucinations differ from text‑only hallucinations: visual hallucination occurs when the model describes nonexistent image content, especially in fine‑grained visual reasoning.

RAG Scenarios

In Retrieval‑Augmented Generation, the pipeline is: parse document → chunk → embed → retrieve → LLM generate.

Parsing quality directly affects retrieval (bad chunks produce poor embeddings) and generation (incomplete or mis‑aligned chunks lead to wrong answers).

Toolchain‑Only Issues

Parsing errors produce unreadable or truncated text, breaking retrieval.

Incorrect table layouts or missing chart semantics corrupt the context fed to the LLM.

Multimodal Integration Strategies

Multimodal embedding : Use CLIP‑style models to embed images and text into a shared space, allowing image queries to be retrieved by text.

Hybrid fallback : Apply the toolchain first; when confidence is low (e.g., scanned pages, complex charts), invoke a multimodal model for a second pass.

When to Prefer Each Approach

LLM + toolchain is ideal for well‑structured, source‑controlled documents, precise field extraction, high‑volume low‑cost processing, auditability, and environments without access to large multimodal models.

Multimodal shines on messy, diverse sources, documents where charts are core information, need to understand image‑text relationships, handwritten content, or when deep comprehension outweighs exact extraction.

Hybrid Architecture Flow

The hybrid design keeps costs low by routing well‑structured documents through the toolchain while delegating only the challenging visual parts to multimodal inference.

Conclusion

Cost : Image tokens are far more expensive than text tokens (5‑10× for mixed PDFs).

Visual hallucination : Still a reliability issue for fine‑grained visual tasks.

Architectural evolution : Leading models (Qwen‑3.5, LLaMA 4, Gemini 3) are turning visual capability from a plug‑in into a native feature.

Long video and long document handling : Remain weak points due to context‑window and inference‑cost limits.

On the principle level, LLMs process token sequences; multimodal models align other modalities to the same vector space via encoders and projection layers, expanding the input channel width. On the engineering level, the trade‑off is between parsing‑induced information loss and multimodal inference cost/controllability. No single solution dominates; the optimal choice depends on document type, accuracy requirements, and budget.