How AI Finds Evidence in Hundred-Page Documents: Multimodal Retrieval Heads in Long-Context VLMs

As Office AI and document agents advance, large models now process entire financial reports, contracts, presentations, and research papers spanning hundreds of pages. However, ingesting a full document does not guarantee the model can correctly use its information. Faced with hundreds of pages of text, images, tables, and charts, the model must first answer a fundamental question: where exactly is the evidence relevant to the current question?

An EMNLP 2026 paper investigates this process from inside the model. The paper, titled "Can Retrieval Heads See Images? Multimodal Retrieval Heads in Long-Context Vision-Language Models" (arXiv:2605.27243), examines whether specific attention heads in long-context vision-language models (VLMs) are responsible for directing the question toward relevant textual or visual content when that evidence is already present in the input.

Identifying Multimodal Retrieval Heads (MMRetHeads)

Inspired by QRHead, the authors propose the MMRetHeads detection method. For each attention head, they analyze the question-to-evidence attention — the attention from question tokens to annotated evidence regions. If the evidence is text, it corresponds to text tokens; if it lies in an image, it corresponds to visual tokens. The stronger the attention pointing to the evidence, the higher the head's retrieval score.

The study covers six long-context VLMs, including Qwen3-VL and Gemma3, across text retrieval, image retrieval, rendered-text retrieval, and same-image retrieval tasks, testing context lengths of 8K, 16K, 32K, 64K, and 128K. Analysis reveals that text and image retrieval share a subset of attention heads, but different context lengths and evidence modalities also recruit distinct heads.

Causal Validation via Head Ablation

Attention pointing to evidence only shows correlation. To test whether the model truly relies on these heads, the team directly masked the highest-scoring retrieval heads and compared the effect against masking an equal number of random heads.

On text and image retrieval tasks, masking retrieval heads caused significant performance drops across context lengths and evidence positions, while random masking had much smaller impact. The same pattern appears in realistic long-document QA:

MMLongBench-Doc: 48.2 → 5.7

SlideVQA: 71.2 → 8.9

Random masking retained 32.2 and 52.6 points respectively. The much larger decline from masking retrieval heads indicates these heads are not merely attending near evidence but play a causal role in the model's ability to access it.

In multimodal reasoning tasks, masking retrieval heads led to failures such as answering "insufficient information" despite the chart being present, reading incorrect content, or hallucinating non-existent evidence — failure modes that appear only after intervention.

From Internal Mechanism to Document Retrieval

The team further repurposes the evidence signals from these heads for multimodal document retrieval. Given a question, MMRetHeads computes relevance scores for candidate pages or layout regions and ranks them accordingly — no additional retriever training required.

Page-level retrieval finds the whole page containing evidence.

Layout-level retrieval further localizes text blocks, tables, images, or charts within the page.

On MMDocIR, Qwen3-VL-8B achieves page-level Recall@1 of 64.7 , surpassing the strongest reported baseline by 7.7 percentage points, and layout-level Recall@1 of 39.0 , exceeding the best baseline by 6.3 points.