Exploring Multimodal GraphRAG: Combining Document Intelligence, Knowledge Graphs, and Large Models

1. Document‑Intelligence Parsing Technical Chain and Hierarchy Construction

The presentation begins with an overview of document‑intelligence parsing techniques, tracing their evolution from rule‑based templates to PDFParse tools and deep‑learning‑based layout analysis, formula detection, and table extraction.

Three main approaches are described:

OCR‑PIPELINE : Convert PDF to images, perform layout analysis, extract blocks (paragraphs, titles, formulas), apply OCR for text, use table analysis for tables, convert formulas to LaTeX, obtain bounding‑box information, sort reading order, and finally reconstruct the document as markdown.

OCR‑Free : End‑to‑end multimodal large‑model processing that directly outputs markdown without intermediate OCR, though real‑world tests show sub‑optimal performance.

PDF‑2‑TEXT : For editable PDFs, use PDFParser for higher accuracy than OCR‑based models.

Advantages of the OCR‑PIPELINE include rich bounding‑box and layout tag information, modular flexibility, CPU‑offline deployment, and support for scanned documents. Drawbacks are dependence on scene‑specific data, limited accuracy in layout, table, and paragraph merging, and slower speed on CPU due to many modules.

OCR‑Free suffers from lack of region segmentation, no bounding‑box output, high GPU resource consumption, large memory footprint for long texts, hallucination issues, and difficulty handling complex documents.

Table parsing challenges are highlighted: multi‑line, missing‑line, and border‑less tables, with traditional CV methods struggling on varied sizes, low resolution, and cross‑page cases.

Layout analysis is a core target‑detection task defining tags such as body, title, image, image‑title, table, etc. The Shanghai AI Lab’s DocLayout‑YOLO achieves strong generalisation through extensive multi‑scene annotation. A lightweight YOLOv8 model (6.23 MB) was open‑sourced for Chinese papers, English papers, research reports, and textbooks, offering fast inference in vertical scenarios.

Formula‑parsing models built on VisionEncoderDecoder architecture were fine‑tuned with early‑stop to avoid over‑fitting, optimizing ExactMatch and EditDistance metrics. The resulting HDNet achieved a Fair‑CR score of 0.963 with only ~300 M parameters, outperforming larger peers.

Figure extraction and caption linking are addressed by extracting <figure,title>, <figure,reference>, and <figure,boundingbox> metadata, enabling downstream rendering and editing.

Reading‑order reconstruction is critical: early rule‑based methods sort by bounding‑box (XY‑cut) yielding modest results. Semantic approaches like LayoutReader improve ordering but rely heavily on annotated data. The recent end‑to‑end DLAFormer models reading order and layout analysis as a joint relation‑prediction task.

2. Multimodal Graph Index Construction and Retrieval Flow

Multimodal graph indexing consists of preprocessing diverse data sources (text, image, video, audio), extracting features (e.g., NLP pipelines, ViT for images, 3D‑CNN for video, speech‑to‑text for audio), and building a graph with nodes (entities, images, video clips) and edges (temporal, semantic, cross‑modal). The graph is stored in databases such as Neo4j or TigerGraph . Feature embeddings are generated using models like ViT (images) and 3D‑CNN (video) and stored in vector databases (FAISS, Milvus). Cross‑modal alignment aligns image‑text, text‑video, and text‑audio pairs before joint indexing.

3. Multimodal Retrieval and Generation Process

User queries (text‑only or text‑plus‑image) are parsed into multimodal components, then retrieved via sub‑graph matching, vector similarity, or cross‑modal association. Retrieved results are fused, re‑ranked for relevance, and fed to a large model for answer generation. Prompt construction may concatenate multimodal embeddings, and the final answer is post‑processed for citation or source attribution.

4. Knowledge‑Graph Enhancement for Chunk Relations and Fine‑Grained Issues

Traditional RAG suffers from noisy chunk retrieval, poor numeric robustness, isolated chunks, limited ES retrieval, uncertain planning, hallucinations, and low explainability. Knowledge graphs (KG) inject expert knowledge, enriching chunk relevance via entity‑level features and hierarchical relations. Microsoft’s GraphRAG uses KG‑based search summaries to boost chunk connections. When a KG exists, it can serve as an additional retrieval source, providing graph‑based embeddings to complement vector features.

Building high‑quality, up‑datable large‑scale graphs is costly. In the document domain, KG can be defined at three granularities: metadata‑level (document titles, topics), chunk‑level (titles, paragraphs, tables, images with parent‑child, co‑occurrence, similarity edges), and entity‑level (named entities and relation‑keyword networks).

Typical application paradigms include:

KG‑enhanced prompt

HiQA (hierarchical chunk recall)

LinkedIn KG‑RAG (dual‑embedding index)

UniQA‑Text2Cypher (KG‑RAG)

HippoRAG (entity‑specific framework)

GRAG (topology‑aware)

Microsoft GraphRAG (KG extraction → community summary)

KAG (full KG integration into RAG)

Comparative analysis:

RAG : Simple chunk‑vector retrieval, low precision and logical coherence.

GraphRAG : Extracts entity relations and community summaries, improving semantic linkage but suffers from noisy graph construction and limited logic.

KG‑QA : Classic pipeline with query parsing, entity linking, semantic reasoning, and source citation; high accuracy and logical soundness but high graph‑building cost and potential knowledge loss.

5. Summary of Key Takeaways

Corpus processing is the dominant factor in RAG performance; its quality directly determines QA effectiveness.

Multimodal large models open new possibilities for end‑to‑end document handling, yet resource constraints often favour pipeline‑based solutions.

Deep document mining still requires human verification to ensure trustworthy results.

Document intelligence remains a challenging long‑tail problem despite renewed interest from large‑model research.

Knowledge graphs must evolve to be lighter, more granular, and structurally richer while retaining their core benefits.

In resource‑limited, text‑dense scenarios, traditional NLP/CV/BERT‑style small models still hold practical value.

6. Q&A Session

Q1: How to choose an appropriate solution for real‑world deployment? The answer emphasizes that multimodal large models (e.g., MMOCR) demand massive resources and extensive training data, and they still exhibit hallucinations. For complex layouts, pipeline approaches (OCR‑PIPELINE + downstream models) are recommended when resources are limited.

Q2: How to improve inaccurate title recognition? Title accuracy depends on layout‑tag definitions. Recommendations include unifying title formats, applying rule‑based post‑processing, and training a semantic model with sufficient data to capture hierarchical cues, though perfect accuracy is hard to guarantee.

Overall, the session concludes with thanks to the audience.