Long-Document VQA Survey: Why Local Relevance Isn't Sufficient Evidence

Introduction

The paper A Task-Centric Survey of Long-Document Visual Question Answering: Taxonomy, Evolution, and Future Directions (arXiv:2610.31385, Oct 2026) by Yufei Tong, Guanjie Cheng, et al. from Zhejiang University and BWTON Technology surveys over 150 papers. It reframes long-document VQA as an evidence process under budget constraints: parsing preserves content and structure, retrieval gathers complementary support, generation integrates and verifies, and evaluation diagnoses reliability and cost.

Task Formulation and Decomposition

Inputs are a question and candidate visual documents; the system outputs an answer with optional evidence sources under context, compute, and tool-call budgets. Closed-domain specifies documents; open-domain requires identifying relevant sources from a shared corpus. Long-document difficulties fall into three categories: input and search scale, visual complexity, and evidence dispersion and dependency.

The authors decompose systems into four interacting components:

Document Parsing – preserves details and structure.

Evidence Organization and Retrieval – selects sufficient support.

Generation Organization and Learning – integrates, verifies, and identifies gaps.

Evaluation – checks answer, evidence, cost, and reliability.

Two main routes exist: direct reading (hierarchical encoding, cross-page fusion, compression) and explicit retrieval (select relevant documents/pages/elements then generate). Explicit retrieval accounts for ~80% of surveyed methods. Three often-confused boundaries are clarified: multi-agent (runtime role collaboration), multi-step reasoning (intermediate feedback revises judgment), and iterative retrieval (feedback acquires new evidence beyond current set).

Taxonomy and Technical Evolution

3.1 Document Parsing

Parsing converts raw pages into accessible content and structure. Two independent axes: external text extraction (OCR, PDF text extraction, VLM transcription/description) and deep parsing (summaries, section trees, relation graphs). Merely describing a chart as text does not count as deep parsing. Extraction-based systems enable text indexing but may lose layout, text-image correspondence, and table relationships; direct page encoding reduces textification loss but remains limited by resolution and visual reading ability. Hybrid approaches use both extracted text and page images.

Evolution: 2021–2023 emphasized readable content + visual layout; 2024 pushed for original-page fidelity driven by explicit retrieval; 2025 introduced summaries, trees, graphs for navigable relations; 2026 moves toward on-demand, question- and process-guided parsing to reduce irrelevant processing. Notable models: LayoutLMv3 (joint text, layout, image), Donut, Pix2Struct (page-to-structure), DSE, ColPali, VisRAG (page representations), SimpleDoc, DocAgent (summaries/hierarchies), AgenticOCR (on-demand region parsing).

3.2 Evidence Organization and Retrieval

Compared across four dimensions: matching modality (text, visual, hybrid), evidence granularity (document, page, element), retrieval mechanism (lexical, vector, multimodal matching, LLM selection/reranking/region refinement), and acquisition mode (fixed process vs. feedback-driven expansion).

Granularity vs. context: Full pages preserve background but add noise; fine-grained regions highlight details but may sever semantics. Table cells without headers, units, and periods are often "relevant but insufficient." Retrieved form may differ from generation input (e.g., text-block match mapped back to source page for visual reading).

Mechanism and timing: ColPali uses multi-vector late interaction; DSE uses page vectors; structure navigation and relation expansion find complementary evidence. Fixed multi-stage retrieval, reranking, and region refinement do not become iterative retrieval unless generation/control feedback drives new evidence acquisition.

Evolution: Early work predicted answer pages; 2024 visual indexing mitigated page overload; 2025 structure, region, and feedback expanded coverage beyond isolated matches; 2026 focuses on cross-page context, evidence state, and marginal value of continued search. DocMemo maintains page beliefs and question memory; budget-allocation methods estimate gain from further retrieval per sub-question. Retrieving more does not guarantee covering necessary dependencies and may increase redundancy and noise. The stopping criterion should be sufficient support for the question, not a fixed page count or highest similarity.

3.3 Generation and Learning

Generators must integrate evidence while controlling context and compute cost. Direct full-document reading preserves context but is expensive; reading retrieved results reduces input but faces missing and noisy evidence. Single-agent vs. multi-role collaboration is an organizational dimension; single-pass vs. feedback loops is a reasoning dimension – they should be described separately. A long chain-of-thought, structured output, or fixed multi-role pipeline can still be single-pass; only when intermediate observations, tools, or verification repeatedly revise judgments and actions does it meet the paper's multi-step reasoning definition. Re-reading and verifying the same evidence set can produce feedback without new retrieval.

Learning objectives: Pre-training builds document representations and modality alignment; supervised fine-tuning uses answer, source, or action demonstrations; reinforcement learning uses answer, evidence, or process rewards. A system using RL does not imply all modules update together – the appendix explicitly distinguishes training scope for retriever, controller, tools, and final answerer.

Evolution: 2021–2023 built document skills; 2024 handled long inputs via cross-page communication, compression, post-retrieval reading; 2025 learned evidence usage, re-reading, tool actions; 2026 focuses on cross-turn state to reduce context accumulation and forgetting. Hi-VT5, GRAM, mPLUG-DocOwl2, VRAG-RL, SCORE exemplify responses to different bottlenecks, not a unified leaderboard. Evidence memory should record what was read, what gaps remain, and where judgments came from. Compression aids continuous reasoning but may cement early misreads; sometimes returning to source pages is necessary rather than having multiple agents reconfirm the same erroneous summary.

3.4 Benchmarks and Evaluation

Evaluation dimensions: answer correctness, retrieval recall/ranking, spatial grounding, efficiency, robustness. Grounding checks predicted region alignment with annotations; efficiency covers parsing, indexing, online inference; robustness tests visual degradation and distractors.

Benchmark evolution: DocVQA, InfographicVQA, ChartQA focus on single-page visual understanding; MP-DocVQA, SlideVQA, DUDE extend to multi-page localization, integration, and unanswerable questions; MMLongBench-Doc, M-LongDoc increase long-input pressure; ViDoRe, MMDocIR target retrieval and grounding; DocScope further checks page, region, fact, answer with controlled conditions to diagnose error sources.

A single average answer score can mask error sources. Correct answers may stem from priors, guesses, or incomplete evidence; good retrieval can be negated by later misreading. Therefore, answer and evidence must be jointly reported, using known-correct evidence, missing evidence, distractor pages to isolate parsing, retrieval, and reasoning bottlenecks. Cross-method comparison should fix candidate documents, input conditions, budget, and evaluation protocol. Open-domain full-corpus page count and specified single-document average page count are not on the same scale; single-page evidence tasks cannot substitute true cross-page integration tests.

Research Trends and Future Directions

4.1 Cross-Component Co-Evolution

Research did not switch abruptly from text to visual to agentic routes. Instead, priorities continuously expanded: first understand content, then acquire high-fidelity sufficient evidence at scale, then continuously maintain and verify that evidence. Component bottlenecks propagate: retrieval reduces input but causes omissions, driving structure-aware acquisition and local parsing; feedback makes parsing and retrieval schedulable actions, requiring generators to maintain state and decide re-read, supplement, or stop; evaluation must track whether these actions actually improve answer support and quality-cost trade-offs.

A typical failure chain: local transcription misses units, summary writes wrong value into memory, subsequent retrieval selects pages based on erroneous summary, generator produces plausible but unsupported comparison. Simply scaling retriever or answer model may not fix upstream errors.

4.2 Challenges and Future Directions

From local relevance to evidence sufficiency. Precise pages/regions don't guarantee all required dependencies. Systems should enumerate required facts, comparison objects, computation conditions, dependencies per question, distinguishing background that aids localization from evidence that directly supports the answer. Example: financial comparison needs both values plus line items, units, periods, calibrations, footnotes; cross-page continued tables and legends should be jointly selected. Future training/test sets should include complete, partially missing, and highly relevant but insufficient evidence sets to test if systems can articulate what's missing and convert that into next search targets.

From correct answer to correctable evidence chain. Correct answers can coexist with wrong intermediate facts or incomplete support. Evidence memory must retain source document, page, region, version, dependencies; on conflict, re-read, retract erroneous records, update downstream judgments – not just change the final answer. Controlled deletion/replacement of key evidence tests whether system detects support loss or fact change; changing irrelevant evidence should keep answer stable. Source citation accuracy, fact reading accuracy, and evidence sufficiency are three distinct checks. More complete evaluation should test system's ability to choose re-parsing, further acquisition, revision, or refusal.

From local acceleration to budget-aware joint coordination. Savings from small-region parsing, few retrievals, or visual compression may be offset by auxiliary model calls, interaction rounds, state maintenance. Search, close reading, re-parsing, verification, and stopping should be decided under a unified quality-cost objective. Future strategies need to estimate marginal gain of an extra action, allocate resolution, parsing scope, retrieval depth, reasoning rounds based on question difficulty, evidence gaps, remaining budget; also judge when direct reading beats indexing first. Reports should cover preprocessing, indexing, auxiliary models, online inference, and show curves across lengths, evidence dispersion, budgets.

Cross-question memory reuse. Follow-up questions often reuse table locations, statistical conventions, verified results. Systems should link dialogue to source pages, reuse still-valid evidence, supplement new info, not reprocess from scratch. After user corrections, document version changes, or condition switches, check if historical memory still applies and revise dependencies.

Cross-task strategy improvement. Remembering where a page is differs from learning better retrieval, reading, verification processes. Inspired by recursive self-improvement, the paper proposes distilling strategies from successes, failures, feedback; validate on held-out tasks, then update skill library or control programs. Example: "before comparing financial values, check units and reporting periods." This is a future direction, not a demonstrated general self-improving system. Key evidence must specify strategy applicability conditions, transfer gains, independent validation – not treat repeated use of same-document info as learning progress.

Conclusion

Long-document VQA is reframed as an evidence process under budget constraints. Research focus evolved from multimodal content understanding to sufficient, high-fidelity evidence acquisition, to state maintenance and continuous use. The next phase must jointly address evidence sufficiency, traceable correction, and budget coordination, while exploring cross-question reuse and cross-task strategy transfer. The goal is not to read as much as possible nor retrieve as little as possible, but to support answers with verifiable evidence at reasonable cost.

Limitations: The survey focuses on long-document QA; text/layout recognition, field/entity extraction are discussed only when they affect QA, not covering all document understanding foundations. Methods use different settings; taxonomy and temporal analysis do not constitute controlled causal proof.

Paper: https://arxiv.org/abs/2610.31385 | PDF: https://arxiv.org/pdf/2610.31385 | Project: https://github.com/yufeiTongZJU/Long-Document-VQA-Survey

Code example

来源：专知
本文
约5500字
，建议阅读
15
分钟
局部相关不等于证据充分，答案正确也不等于过程可信。