Why Visually‑Rich Document Understanding Looks Like High‑End Docs: A Static Multimodal Overview

Recent advances show that Chinese researchers have long led traditional OCR, with figures like Dr. Sun Jian influencing major companies such as Megvii, SenseTime, Baidu, Tencent, and ByteDance. Building on this foundation, Visually‑Rich Document Understanding (VRDU) has emerged as a new frontier, exemplified by the AAAI‑2025 VRDU competition and rapid open‑source releases from Baidu and ByteDance.

The multimodal landscape includes pioneering models such as DCGAN, CLIP, DALL·E, Whisper, Diffusion, LDM, U‑Net, ViT, OpenVLA, VMC‑World Models, and JEPA, most of which are driven by foreign research groups. Although the DiT model has Chinese authors, overall original multimodal model ideas remain scarce, with Chinese contributions largely focused on improvements to existing architectures.

In the VRDU domain, the LayoutLM series represents the first truly original work from Chinese researchers. LayoutLM (2019) introduced text plus 2‑D positional embeddings; LayoutLMv2 (2020) added visual features; LayoutLMv3 (2022) unified text and image representations. The seminal paper "Pre‑training of Text and Layout for Document Image Understanding" demonstrated the necessity of joint layout‑text pre‑training, launching the modern VRDU field.

LayoutLM models have been the top‑ranked solutions in the AAAI‑2025 VRDU competition, especially for the two tracks: (A) key‑information extraction from forms and (B) key‑information localization. The winning rb‑ai team segmented form images into top, middle, and bottom regions using YOLOv8, then fine‑tuned LayoutLMv3 on each region and applied heuristic post‑processing to resolve mis‑classifications.

VRDU documents fall into three categories—parseable PDFs, printed pages, and handwritten scans—and support several core tasks: key‑information extraction (KIE), document layout analysis (DLA), document question answering (DQA), and table structure recognition (TSR). Typical hard sub‑tasks include flow‑chart recognition, wireframe data extraction, icon semantics, border‑less table alignment, embedded‑image text alignment, cross‑document information alignment, and reinforcement‑learning‑based fine‑tuning.

Methodologically, VRDU approaches can be grouped by whether they rely on traditional OCR input and by the modalities they fuse (text, block‑image, global layout, multimodal). Two major model families exist: (1) multi‑task pre‑trained models that jointly capture text, layout, and visual features at token‑, block‑, and page‑level granularity, often using encoder‑decoder designs for image‑text generation; (2) non‑pre‑trained frameworks that treat each modality separately.

Interaction patterns between text and vision include (1) self‑attention over multimodal tokens (state‑of‑the‑art), (2) cross‑attention, and (3) early ROI‑pooling methods. Core architectures evaluated for VRDU include ViT (2020), DETR (2020), ViTDet (2022), DiT (2022), and NaViT (2023). ViT serves as a low‑cost visual backbone for layout tasks; DETR offers end‑to‑end detection without post‑processing; ViTDet combines ViT with detection heads; DiT is suited for synthetic data generation; NaViT best handles high‑resolution, complex layouts and integrates well with large multimodal models such as DocLLM.

Open‑source toolkits illustrate these design choices. MinerU (Shanghai AI Lab) follows a two‑stage pipeline: coarse‑grained layout analysis followed by fine‑grained content recognition, supporting LaTeX formula detection, complex table parsing, and even molecular structure recognition. PaddleOCR‑VL (PaddlePaddle) also adopts a two‑stage approach: PP‑DocLayoutV2 predicts page‑level regions and reading order, then PaddleOCR‑VL‑0.9B performs fine‑grained recognition of text, tables, formulas, and charts, achieving higher accuracy and efficiency by avoiding full‑resolution VLM computation. DeepSeek‑OCR removes the traditional OCR front‑end, using a DeepSeek‑3B decoder to improve long‑document processing efficiency. Dots.ocr (rednote‑hilab) encodes documents with ViT and decodes with QWen, delivering comparable performance to MinerU and PaddleOCR‑VL.

Overall, open‑source VRDU solutions combine ViT, DETR, ViTDet, and NaViT components, but differ in OCR reliance and stage separation. Given PaddleOCR’s strong industrial reputation, the article recommends PaddleOCR‑VL as the preferred baseline for most VRDU applications.