Raggae: Building RAG Pipelines for Complex Documents via Layout Analysis

When humans read documents, they naturally associate prices with table items, link captions to images, and distinguish paragraphs in multi‑column layouts. These spatial relationships aid comprehension, but typical document processing flattens everything into a single string before chunking, losing the original structure. If a RAG system ingests a document where prices detach from item names, it may retrieve the right page but feed the model incorrect evidence.

Layout analysis finds content regions and their positions

RAG pipelines can use this layout map to route paragraphs, tables, and images to dedicated readers, then reassemble context based on positional relationships instead of relying on the file’s internal byte order. The article walks through concrete tasks — table and figure localization, reading‑order inference, and repeated header/footer detection — using the Python SDK Raggae, which splits document processing into independent workers.

pip install raggae

PDF with columns, tables, and figures

The example file complex-report.pdf (two pages) mixes two‑column body text, a cross‑column table, and a chart with a caption on the second page. Flattening scrambles column order, breaks table‑header alignment, and may miss the chart entirely. Sending all content to both text and table readers would duplicate the table. NativePdfLayoutAnalyzer uses PDF‑internal character, line, and raster image positions to split text into candidate blocks, detect bordered table regions, and record chart image coordinates.

from raggae.documents.layout_analysis import NativePdfLayoutAnalyzer

pdf_worker = NativePdfLayoutAnalyzer()
pdf_layout = pdf_worker.analyze("complex-report.pdf")
print(pdf_layout.outcome)
print("pages:", len(pdf_layout.surfaces))
print("regions:", dict(Counter(region.kind for region in pdf_layout.regions)))
for region in pdf_layout.regions:
    if region.kind in ("table", "image"):
        print(region.id, region.kind, region.bbox)

Output:

success
pages: 2
regions: {'text': 22, 'table': 1, 'image': 1}
page-1-r8 table (48.0, 312.0, 564.0, 420.0)
page-2-r5 image (48.0, 180.0, 564.0, 402.0)

Table cell text is excluded from regular text regions, so text and table paths never double‑extract the same content. DocumentRegionRenderer can draw region boxes on the original pages for visual verification (requires pip install 'raggae[layout-visual]'). Blue boxes = text, green = tables, orange = images.

Which region comes next? Reading‑order resolution

Bounding boxes only say where content sits; to read the report correctly we need the sequence between two‑column text, cross‑column table, and next‑page image. GeometricReadingOrderResolver builds precedence links from the existing layout result by analyzing gaps between rectangles — no re‑parsing needed.

from raggae.documents.layout_analysis import GeometricReadingOrderResolver

resolver = GeometricReadingOrderResolver()
order = resolver.resolve_order(pdf_layout, direction="ltr")
print(order.outcome, "links:", len(order.edges))
print("unresolved:", len(order.unresolved_region_ids))
for edge in order.edges:
    if edge.before_region_id in ("page-1-r8", "page-2-r5"):
        print(edge.before_region_id, "->", edge.after_region_id, edge.basis)

Result:

success links: 23
unresolved: 0
page-1-r8 -> page-1-r9 horizontal_gap
page-2-r5 -> page-2-r6 horizontal_gap

The resolver suggests reading the image region page-2-r5 before the nearby text page-2-r6 based on horizontal gap. This infers sequence, not that the text is a caption. DocumentReadingOrderRenderer visualizes these links with arrows and page‑level reading numbers.

Common PDF layout patterns handled

Procurement table between paragraphs — Raggae isolates the table, preserving surrounding paragraph boundaries so downstream readers see item, quantity, and amount columns together.

Three‑column body under a full‑width intro — Columns are grouped separately; intro and closing each get their own region, preventing interleaving.

Rotated pages — Coordinates are computed in the displayed orientation, aligning region boxes with actual visible text and tables.

Two figures with nearby captions — Each image gets a distinct region; adjacent caption text is also located for later association.

What the layout map records

All content‑discovery workers return a LayoutAnalysisResult with: surfaces — content carriers (PDF pages, slides, worksheets, image frames, continuous body). regions — identified content areas, each with an ID, type, and location in the source file. bbox — rectangle (left, top, right, bottom) in points for PDF, pixels for images. findings — notes on coverage gaps, processing limits, or issues.

The map stores regions and positions, not extracted paragraph text or image files. Region IDs do not encode reading order. To let later readers locate original elements, the layout map and source file must be saved together.

Word report with nested tables and embedded images

receiving-report.docx

contains a purchase table on page one; page two has a reference table with a nested sub‑table on the left and an embedded image on the right. Traversing only top‑level paragraphs misses table‑internal content; recursing into every cell paragraph duplicates extraction. DocxLayoutAnalyzer follows the document’s structured XML, locating elements by their XML path without guessing from rendered pixels.

from raggae.documents.layout_analysis import DocxLayoutAnalyzer

analyzer = DocxLayoutAnalyzer()
word_layout = analyzer.analyze("receiving-report.docx")
print(word_layout.outcome)
print(dict(Counter(region.kind for region in word_layout.regions)))
for region in word_layout.regions:
    if region.kind in ("table", "image"):
        print(region.id, region.kind, region.location.child_indexes)

Result:

success
{'heading': 2, 'text': 3, 'table': 2, 'image': 1}
body-r3 table (2,)
body-r6 table (6,)
body-r7 image (6, 2, 1, 2, 0, 0)

Each top‑level table stays a single region; nested cell paragraphs and sub‑tables are not promoted to body regions. The image path drills into the table’s child elements. Location data includes the internal part name word/document.xml, so other readers can address the exact same XML node. The analyzer treats the Word body as a continuous flow whose element paths cross explicit page breaks, preserving document structure.

Other Word scenarios

Custom heading styles in an employee handbook — recognized as heading to keep chapter boundaries.

Reference tables with embedded images — both table and image retain precise locations for separate readers.

Reports with repeated phrasing — element paths point to original paragraphs/tables, avoiding deduplication by text search.

Slide deck where shape z‑order ≠ reading order

A shipment briefing slide contains title, intro, inventory table, image, and a note below the image. Extracting text alone loses the table structure and image. Reading shapes in file storage order is unreliable because stacking order ≠ presentation reading order. PptxLayoutAnalyzer returns both the shape’s source path and its slide coordinates.

from raggae.documents.layout_analysis import PptxLayoutAnalyzer

analyzer = PptxLayoutAnalyzer()
slide_layout = analyzer.analyze("shipment-briefing.pptx")
print(slide_layout.outcome)
print("slides:", len(slide_layout.surfaces))
print(dict(Counter(region.kind for region in slide_layout.regions)))
for region in slide_layout.regions:
    if region.kind in ("table", "image"):
        print(region.id, region.kind, region.bbox)

Result:

success
slides: 2
{'heading': 1, 'text': 4, 'table': 1, 'image': 1}
slide-1-s2 table (40.0, 200.0, 340.0, 340.0)
slide-1-s3 image (400.0, 200.0, 580.0, 320.0)

Each region records the source slide part and shape path (e.g., (3, 1) = fourth shape, second child). Coordinates are relative to the slide top‑left; reading‑order suggestions are stored separately.

Other presentation scenarios

Grouped, scaled, or mirrored shapes — nested transforms are resolved to final slide coordinates; only leaf content regions are kept.

Lower text box created before upper one — geometric resolver proposes reading upper first, ignoring creation time.

Multi‑slide decks — each slide is its own surface; tables and images on different slides stay separated.

Workbook with multiple independent content areas

A warehouse sheet may have inventory on the left, costs on the right, and a merged‑cell label below. Treating the whole used range as one table mixes three areas and empty columns. XlsxLayoutAnalyzer splits them.

from raggae.documents.layout_analysis import XlsxLayoutAnalyzer

analyzer = XlsxLayoutAnalyzer()
sheet_layout = analyzer.analyze("stock-and-costs.xlsx")
print(sheet_layout.outcome)
print(dict(Counter(region.kind for region in sheet_layout.regions)))
for region in sheet_layout.regions:
    print(region.kind, region.location.sheet_name, region.location.cell_range)

Result:

success
{'table': 1, 'unknown': 5}
table Warehouse A1:B4
unknown Warehouse A7:D7
unknown Warehouse F2:G4
unknown No declared table A1:B3
unknown No declared table F1
unknown No declared table G2

Declared tables keep table type; other content areas become unknown. Each region records its cell range for downstream readers. Inventory, costs, and merged label are separated; diagonal labels stay independent. Sheet names prevent address collisions across sheets.

Other spreadsheet scenarios

Logo anchored at E6 — image region logs sheet, image index, and anchor cell.

Sparse sheet with values only at A1 (0) and last cell (False) — only two non‑empty regions returned; formatted‑only cells ignored.

Workbook with blank sheets — empty sheets retain their surface but generate no content regions, preserving workbook structure.

Worksheets use cell ranges and drawing references, not page rectangles.

Web article where structure lives in HTML tags

Saved pages already mark headings, body, tables, and captions via HTML tags. Stripping tags early discards usable structure. HtmlLayoutAnalyzer inspects elements directly — no image fetching, no script execution.

from raggae.documents.layout_analysis import HtmlLayoutAnalyzer

analyzer = HtmlLayoutAnalyzer()
html_layout = analyzer.analyze("warehouse-report.html")
print(html_layout.outcome)
for region in html_layout.regions:
    print(region.kind, region.location.tag, region.location.child_indexes)

Result:

success
heading h1 (0, 1, 0, 0)
text p (0, 1, 0, 1)
heading h2 (0, 1, 0, 2, 0)
table table (0, 1, 0, 2, 1)
image img (0, 1, 0, 3, 0)
caption figcaption (0, 1, 0, 3, 1)
text p (0, 1, 0, 4)

Table and its native <caption> stay together; <figcaption> is separate for later association. Element paths start from the parsed document root, counting child nodes — independent of screen coordinates. Readers can locate the table via its path without re‑scanning the page.

Other HTML scenarios

Multi‑level headings — heading tags preserve main vs. section titles; paragraphs become text regions.

Tables containing paragraphs and images — whole table is one region; nested images can have independent locations for image readers.

HTML fragments (not full pages) — analyzed directly; paths still start from the fragment’s root.

Scanned report where every page is a single image

Scanned PDFs look like normal PDFs but store text, tables, charts as pixels in one big image per page. Native analyzers find no selectable text boxes or table lines. OCR recognizes characters but cannot decide which regions go to a table reader vs. a chart reader. Visual layout detection is needed to segment regions. VisualLayoutAnalyzer renders PDF pages or opens image files, sends pixels to a local detection service (e.g., Paddle layout detector or Docling Heron model), receives categorized bounding boxes, and converts coordinates to the document’s coordinate system.

pip install 'raggae[layout-visual,layout-docling]'
# Alternative: pip install 'raggae[layout-visual,layout-paddle]'

Model weights must be downloaded beforehand. Environment variable RAGGAE_DOCLING_MODEL_DIR points to the local Heron model directory; instantiation does not trigger downloads. Inference runs locally.

import os
from raggae.documents.layout_analysis import (
    DoclingLayoutService, VisualLayoutAnalyzer,
)

service = DoclingLayoutService(
    model_path=Path(os.environ["RAGGAE_DOCLING_MODEL_DIR"]),
    device="cpu",
)
visual_worker = VisualLayoutAnalyzer(service=service)

scan_pdf = Path("scanned-report.pdf")
visual_layout = visual_worker.analyze(scan_pdf)
print(visual_layout.outcome)
for surface in visual_layout.surfaces:
    kinds = Counter(
        region.kind for region in visual_layout.regions
        if region.surface_id == surface.id_
    )
    print(surface.id_, dict(kinds))

Result:

success
page-1 {'table': 1, 'text': 5, 'image': 1, 'caption': 1, 'heading': 1}
page-2 {'text': 5, 'table': 1, 'image': 1, 'caption': 1, 'heading': 1}

Both pages yield candidate text, table, and image regions; the page without native text still enables downstream extraction. Swapping to Paddle only requires replacing the injected detection service — worker interface and output structure stay identical, so downstream readers and renderers continue working.

from raggae.documents.layout_analysis import PaddleLayoutService

paddle_service = PaddleLayoutService(
    model_path=Path(os.environ["RAGGAE_PADDLE_MODEL_DIR"]),
    device="cpu",
)
paddle_layout = VisualLayoutAnalyzer(service=paddle_service).analyze(scan_pdf)

Other visual analysis scenarios

PNG receipts or reports — same worker, no need to wrap images in PDF.

Multi‑frame TIFF or mixed native/scanned PDF — analyzed per frame/page; visual detection is optional and not automatically run on every PDF.

Reusing local detection service — initialize once, pass the same service object to multiple analyses to avoid reloading models.

Mis‑joined two‑column body text

Locating all text blocks isn’t enough to determine reading order. Research articles read left column fully before right column; sorting by vertical coordinate alone causes jumping between columns. GeometricReadingOrderResolver infers order from whitespace between regions: vertical gaps often separate columns, horizontal gaps separate headings from body. It returns region precedence suggestions from an existing layout map without reopening the file.

Earlier we used it to keep tables intact and place images before adjacent captions. Original region positions remain unchanged.

Other reading‑order scenarios

Slide where lower text box created first — resolver proposes upper box first based on actual coordinates.

Right‑to‑left columns — pass direction="rtl" to read columns right‑to‑left; convention set by caller.

Wide table between body paragraphs — table stays a single unit in reading order, avoiding cell‑paragraph interleaving.

Multi‑page documents — inter‑page links connect contentful pages for sequential assembly.

Repeated headers/footers that pollute every chunk

A 100‑page handbook may repeat company name, internal‑use notice, and page numbers on every page. Including them in every chunk wastes capacity and breaks continuity, but blanket removal risks deleting intentional repetitions (e.g., "Always record waybill number" appearing in multiple chapters). PdfRepeatedMarginDetector combines text content and relative position across pages to identify true running headers/footers.

from raggae.documents.layout_analysis import PdfRepeatedMarginDetector

analyzer = NativePdfLayoutAnalyzer()
handbook_pdf = Path("handbook.pdf")
handbook_layout = analyzer.analyze(handbook_pdf)
margins = PdfRepeatedMarginDetector().detect(handbook_layout, handbook_pdf)
for group in margins.groups:
    print(group.kind, group.basis, len(group.region_ids))

Result:

header repeated_text 2
footer repeated_text 2
footer page_number 2

The detector reads a small amount of native text from the same PDF used for layout, compares content and cross‑page relative positions, and recognizes patterned page numbers. Groups are saved as candidates only — original text is untouched. Later assembly can ignore, metadata‑tag, or annotate these regions.

Other repeated‑margin scenarios

Changing page numbers and invoice numbers — Page 1 of 2 / Page 2 of 2 recognized as a page‑number group; Invoice 1001 / Invoice 1002 do not match. Fixed labels like Release 2026 still match as normal text.

Reminder repeated in body — same requirement appears mid‑page on both pages, so it falls outside header/footer bands. Repetition and location are judged together.

Different‑sized but similarly positioned headers — position comparison uses proportions relative to each page size, not absolute dimensions.

Keeping charts linked to their explanatory text

A bar chart with four bars has a caption directly below; a distant paragraph discusses something else. Associating the nearby caption with the chart preserves needed context during later assembly. In caption-report.pdf, page one’s caption sits under the chart, page two’s caption sits above a table. RuleBasedRegionRelationshipDetector uses existing layout results to propose caption links based on distance and alignment; for containment relationships it references element structure paths or bounding rectangles.

from raggae.documents.layout_analysis import RuleBasedRegionRelationshipDetector

caption_layout = pdf_worker.analyze("caption-report.pdf")
relationship_worker = RuleBasedRegionRelationshipDetector()
links = relationship_worker.detect(caption_layout)
print(links.outcome)
for link in links.relationships:
    print(link.kind, link.source_region_id, "->", link.target_region_id, link.basis)

Result:

success
caption_candidate page-1-r5 -> page-1-r4 proximity
caption_candidate page-2-r4 -> page-2-r5 proximity

The detector links nearby captions to charts/tables; proximity indicates spatial closeness as the basis, for use in later assembly.

Combining everything in an ingestion pipeline

Previous sections called each worker separately to show individual responsibilities. In a real ingestion flow these steps can be orchestrated together. DocumentLayoutAnalyzer selects the appropriate content‑discovery worker by file extension (or explicit source_format) and runs configured analyses.

from raggae.documents.layout_analysis import DocumentLayoutAnalyzer

report = DocumentLayoutAnalyzer().analyze(handbook_pdf)
print(report.outcome)
print("regions:", len(report.layout.regions))
print("margin groups:", len(report.repeated_margins.groups))

Result:

success
regions: 16
margin groups: 3

The final report contains the raw layout plus independent reading_order, relationships, and repeated_margins — each computed against the same layout map. Detecting headers does not silently delete header regions before reading‑order resolution.

Default analyzer selection by extension; for raw bytes use source_format="pdf". To only get positions, disable later stages:

discovery_only = DocumentLayoutAnalyzer().analyze(
    handbook_pdf,
    reading_order=False,
    relationships=False,
    repeated_margins=False,
)

For image files, provide visual_service=service. For scanned PDFs with visual detection, inject the analyzer directly:

scan_report = DocumentLayoutAnalyzer(
    analyzer=VisualLayoutAnalyzer(service=service),
).analyze(scan_pdf, repeated_margins=False)

Table boundaries, paragraph positions, slide shape coordinates, and potential caption‑object links cannot be fully preserved in a plain‑text string. Layout analysis records this information. Downstream readers then specialize: text readers extract from text regions, table readers from table regions, image readers from image regions, and positional data reassembles the results. Subsequent stages extract actual content, assemble using reading order and region relationships, and place context into retrievable chunks.

Locate evidence first, extract content next, then organize semantically. Preserving document structure at ingestion makes it easier for RAG systems to find the correct supporting evidence when answering questions.

Author: Okan Yenigün