From Visual Compression to Memory: MonkeyOCRv2 Reconstructs Document Evidence

Why visual encoders matter – In a controlled experiment where every component except the visual encoder was frozen (Qwen3‑1.7B language model, same training data, optimizer settings and decoding pipeline), MonkeyOCRv2‑B achieved 57.2 points on eight document‑understanding benchmarks, out‑performing the strongest baseline OpenVision‑B by 13.2 points. This proves that the visual encoder alone can determine how much reliable evidence reaches the LLM.

Risks of modular design – Treating the visual encoder as a replaceable front‑end (e.g., CLIP, DINO, SAM) hides the fact that once characters, numbers, formulas or layout relations are discarded early, the back‑end can no longer recover them and must rely on language priors. Strong language models may “correct” rare but correct visual tokens, producing fluent but inaccurate answers.

Document‑specific precision – Unlike natural‑image pre‑training that tolerates minor visual changes, document AI requires exact preservation of tiny details (a decimal point, a subscript, a table line). A change from 1.5 mg to 15 mg illustrates how a single visual error can alter meaning by an order of magnitude.

Reconstruction as visual memory – MonkeyOCRv2 jointly trains on image‑to‑text generation and pixel‑level reconstruction. The generation loss cares only about the final text, while the reconstruction loss forces the same visual tokens to retain enough information to rebuild the original page, including strokes, symbols, and layout.

Evidence chain – first link: frozen back‑end

System‑level leaderboards conflate many variables, so MonkeyOCRv2 performed a controlled swap: all visual encoders fed the same frozen Qwen3‑1.7B, with identical downstream data, optimizer and decoding. The only intentional change was the visual encoder itself, confirming that the encoder is an independent variable that can cause a two‑digit performance gap.

Second link: removing language shortcuts

To force reliance on visual evidence, the authors created two text versions: one preserving normal semantics and one with characters randomly shuffled, destroying language context. Images were down‑sampled from a long‑side of 1288 to 448. At the lowest resolution, a model without reconstruction scored 55.4 % accuracy on shuffled text, while adding reconstruction raised it to 72.1 % (+16.7 pp). The gap between normal and shuffled text narrowed from 29.3 pp to 15.3 pp, showing reconstruction helps when language priors are unavailable.

Third link: visual‑language conflict (CHAOS‑Bench)

CHAOS‑Bench deliberately corrupts a few characters on the page, creating a visual token that is clear but semantically implausible. Models that rely on language priors tend to “correct” the error, whereas faithful models output the exact visual token. MonkeyOCRv2‑B‑Parsing achieved a page‑level recall of 17.9, surpassing HunyuanOCR‑1.5 (14.2), DeepSeek‑OCR 2 (6.3) and PaddleOCR‑VL‑1.6 (6.0). An ablation showed that removing reconstruction dropped recall from 14.7 to 12.1, confirming the contribution of the reconstruction objective.

Cross‑task transfer across seven document tasks

MonkeyOCRv2 was evaluated on text recognition, formula recognition, text detection, document tampering detection, overlapped‑text segmentation, document parsing and document understanding. Using a frozen 0.1 B visual encoder with a 0.6 B language model, MonkeyOCRv2‑B‑Parsing scored 83.3 on the 17‑language MDPBench (81.7 on photographed documents, 82.1 on non‑Latin scripts), the highest among open‑source models. Across downstream tasks, it consistently improved performance: +2.3 pp for CRNN, +0.4 pp for near‑saturation PARSeq, and gains of 3.3, 7.5 and 5.3 pp on text detection, tampering detection and overlapped‑text segmentation respectively. Even a 1.1 B‑parameter MonkeyOCRv2‑S outperformed a 3.25 B‑parameter UniMERNet‑B on formula recognition.

Public dataset MonkeyDoc v2

To enable fair, reproducible research, the authors released MonkeyDoc v2, containing 1.13 B document images covering 17 languages, including 8 M page‑level images and 105 M fine‑grained elements (text, tables, formulas). This public corpus allows the community to answer foundational questions about the contributions of image‑to‑text generation versus reconstruction, optimal data ratios, low‑resource language coverage, and the trade‑off between encoder size and pre‑training objectives.

In summary, the combined evidence—controlled encoder swaps, shuffled‑text robustness tests, CHAOS‑Bench conflict analysis, and consistent multi‑task improvements—supports the hypothesis that “reconstruction equals visual memory”: a visual encoder that can faithfully reconstruct its input retains the critical page evidence needed for accurate downstream document AI.