pdf-inspector: 18.3K-Star Rust Library Cuts PDF Processing Costs 50% in 200ms

Problem: Naive OCR Everything Wastes Resources

Most teams send every PDF to OCR, but ~54% of PDFs (reports, papers, invoices, contracts) already contain selectable text layers. Full OCR wastes API fees and GPU cycles, adds 2-10 seconds per page latency, and can introduce hallucinated characters that break document structure.

Solution: pdf-inspector as Intelligent Triage Layer

pdf-inspector is a pure-Rust, zero-ML, zero-cloud-dependency library extracted from Firecrawl's production Fire-PDF engine. It sits at the pipeline entrance, classifies each PDF in 10-50ms, extracts text pages locally in ~150ms (total <200ms), and emits pages_needing_ocr so only image pages go to downstream OCR.

Four Core Capabilities

1. Intelligent PDF Classification

Outputs four categories with 0-1.0 confidence: TextBased (extract locally), Scanned (full OCR), ImageBased (no text), Mixed (per-page routing). Three scan strategies: EarlyExit (default, stops at first non-text page), Full (scan all pages), Sample(n) (random N pages for huge files). Example: 210-page financial report with 150 text pages + 60 scanned pages → only 60 pages sent to OCR.

2. Precise Text Extraction with Coordinates

Every text span includes page X/Y coordinates, font name, and font size. Coordinate engine auto-detects multi-column layouts (e.g., two-column papers) and outputs reading-order-correct text. Built-in CID font (Identity-H Type0) decoder with ToUnicode CMap parsing supports UTF-16BE, UTF-8, Latin-1 fallback and Adobe Glyph List mapping. Pages with decoding issues are tagged encoding issue for automatic OCR fallback.

3. High-Quality Markdown Conversion

Unlike raw-text tools (PyMuPDF, pdfplumber), pdf-inspector preserves semantic structure: heading levels (H1-H4 via font-size clustering threshold 0.5pt), lists (bullet, numeric, alphabetic), code blocks (monospace font detection), rich text (bold, italic, sub/superscript, links), word-hyphenation repair, redundancy filtering (page numbers, TOC dot leaders), and page-break comments <!-- Page N --> for post-processing header/footer removal.

4. Dual-Engine Table Recognition

PDFs lack native table tags. pdf-inspector combines: (1) Rectangle border detection parsing vector drawing commands and union-find cell aggregation; (2) Heuristic alignment detection inferring rows/columns from text coordinate patterns for borderless tables. Handles financial statements, footnoted tables, and cross-page continued tables, outputting standard Markdown tables.

Benchmark Results (OpenDataLoader-Bench, 200 real PDFs, Apple M4 Pro, OCR disabled)

pdf-inspector : overall 0.875, reading order 0.915, tables 0.814, headings 0.788, total time 0.470s

LiteParse : overall 0.873, reading order 0.913, tables 0.693, headings 0.811, total time 0.750s

OpenDataLoader : overall 0.831, reading order 0.902, tables 0.489, headings 0.739, total time 2.569s

PyMuPDF4LLM : overall 0.735, reading order 0.886, tables 0.401, headings 0.424, total time 17.117s

MarkItDown (Microsoft) : overall 0.589, reading order 0.844, tables 0.273, headings 0.000, total time 16.165s

pdf-inspector is ~36x faster than PyMuPDF4LLM and ~34x faster than MarkItDown, leads overall score, reading order, and table extraction; heading detection slightly trails LiteParse but within acceptable range.

Integration Methods

Rust

[dependencies]
pdf-inspector = "0.1"

use pdf_inspector::process_pdf;
fn main() -> Result<(), Box<dyn std::error::Error>> {
    let result = process_pdf("document.pdf")?;
    println!("PDF type: {:?}", result.pdf_type);
    if let Some(markdown) = &result.markdown {
        println!("Markdown:
{}", markdown);
    }
    Ok(())
}

Python

pip install pdf-inspector
# or from source: pip install maturin && maturin develop --release

import pdf_inspector
result = pdf_inspector.process_pdf("document.pdf")
print(result.pdf_type)
print(result.markdown)
print(result.pages_needing_ocr)
# Fast pre-check only
detect = pdf_inspector.detect_pdf("document.pdf")
if detect.pdf_type == "text_based":
    print("Text PDF, extract locally")
else:
    print(f"OCR needed pages: {detect.pages_needing_ocr}")

Node.js / Bun

npm install @firecrawl/pdf-inspector

import { readFileSync } from 'fs';
import { processPdf } from '@firecrawl/pdf-inspector';
const buffer = readFileSync('document.pdf');
const result = processPdf(buffer);
console.log(result.pdfType);
console.log(result.markdown);

Browser WASM (Privacy-First)

npm install @firecrawl/pdf-inspector-wasm

import init, { processPdf } from '@firecrawl/pdf-inspector-wasm';
await init();
const response = await fetch('/document.pdf');
const pdfBytes = new Uint8Array(await response.arrayBuffer());
const result = processPdf(pdfBytes);
console.log(result.pdfType);
console.log(result.markdown);

CLI Tool

cargo install pdf-inspector
pdf2md document.pdf          # full Markdown output
pdf2md document.pdf --json   # full metadata JSON
pdf2md document.pdf --select-pages 1,3,5-10
detect-pdf document.pdf      # type detection only
detect-pdf document.pdf --analyze --json

Recommended RAG Pipeline Architecture

PDF file → pdf-inspector pre-check

Branch 1: TextBased → local Markdown extraction → cleaning → knowledge-base chunking

Branch 2: Mixed → extractable pages to Markdown, pages_needing_ocr pages to OCR → merge results

Branch 3: Scanned / ImageBased → full downstream OCR

Result: 54% of documents bypass expensive OCR entirely, dramatic cost and latency reduction at scale.

Known Limitations

No built-in OCR — scanned pages require external OCR (PaddleOCR, Tesseract, cloud APIs).

Heading detection slightly below LiteParse; occasional mis-leveling.

Complex multi-column edge cases may still misorder text (open issues exist).

Rare CJK custom/encrypted fonts can still cause decoding errors.

Very large PDFs with heavy vector graphics may see slower parsing.

No automatic header/footer stripping; page-break comments provided for custom post-processing.

When to Use / Avoid

Strong fit: RAG knowledge bases, AI agent document QA, high-volume financial/legal/academic PDFs with tables, browser-local parsing for privacy, air-gapped/offline deployments, existing full-OCR pipelines needing cost reduction.

Avoid if: >70% scanned pages, or predominantly non-standard/encrypted fonts and exotic layouts.

Core philosophy: triage first, route later, never call OCR when local extraction suffices. Place pdf-inspector as the first gate in your document pipeline.

Project: https://github.com/firecrawl/pdf-inspector