RAG-Anything: A Universal RAG Framework for PDFs, Office Docs, and Images

When building knowledge bases, data cleaning—especially for PDFs, Word, Excel, PPT files that contain tables, formulas, or screenshots—is often the biggest pain point; most existing RAG solutions handle plain text well but struggle with multimodal content.

RAG-Anything was created to solve this problem. It provides an end‑to‑end multimodal pipeline that can ingest any format—PDF, Office documents, images, or scientific papers with complex equations—and then perform intelligent retrieval.

Core features:

All‑format support : PDF, Word, PPT, Excel, images.

High‑fidelity parsing : integrates MinerU to preserve document structure and avoid garbled tables.

Professional content analysis : dedicated processors for images, tables, and equations, enabling queries such as “What does Figure 3 illustrate?”

Multimodal knowledge graph : extracts entities from both text and images and builds relationships, surpassing pure vector search.

Hybrid intelligent retrieval : combines text and multimodal embeddings for deeper understanding.

Installation is straightforward via the PyPI package:

# Basic install
pip install raganything

# Full install with all format support (recommended)
pip install 'raganything[all]'

Processing Office documents requires LibreOffice. Example commands for macOS and Ubuntu are provided.

Usage (asyncio‑based) demonstrates configuring the engine with the MinerU parser, enabling image, table, and equation processing, initializing the RAG engine, processing a complex PDF, and performing a hybrid query that combines text and image information.

import asyncio
from raganything import RAGAnything, RAGAnythingConfig
from lightrag.llm.openai import openai_complete_if_cache, openai_embed
from lightrag.utils import EmbeddingFunc

async def main():
    api_key = "sk-xxxxxxxx"
    config = RAGAnythingConfig(
        working_dir="./rag_storage",
        parser="mineru",
        enable_image_processing=True,
        enable_table_processing=True,
        enable_equation_processing=True,
    )
    # LLM and embedding functions omitted for brevity
    rag = RAGAnything(
        config=config,
        llm_model_func=llm_model_func,
        vision_model_func=vision_model_func,
        embedding_func=embedding_func,
    )
    await rag.process_document_complete(
        file_path="./my_complex_paper.pdf",
        output_dir="./output",
        parse_method="auto",
    )
    result = await rag.aquery(
        "What does Figure 2 illustrate? Analyze the table data as well.",
        mode="hybrid",
    )
    print("Answer:", result)

if __name__ == "__main__":
    asyncio.run(main())

Design mechanisms for heterogeneous data:

Divide and conquer : separate pipelines handle images and text.

Multimodal alignment : visual information is converted into vectors aligned with text vectors in a shared space.

Knowledge‑graph enhancement : builds logical relations between “graph” and “text” entities.

Potential pitfalls:

Resource consumption : MinerU and multimodal models (e.g., GPT‑4o or open‑source VLMs) require significant GPU memory or API quota.

Dependency complexity : LibreOffice and numerous Python libraries (especially those pulled by MinerU) may complicate environment setup; Docker or Conda is recommended.

Conclusion: RAG-Anything aims to unify multimodal heterogeneous data processing for enterprise‑grade knowledge bases. While it is not the lightest solution, it currently offers one of the most comprehensive open‑source approaches to handling complex documents.