Why pdf-inspector Has Earned 11.7k Stars on AI‑Heavy GitHub

Problem

Many RAG and document‑processing pipelines send every PDF to OCR, even though more than half of PDFs already contain a native text layer. OCR is slow and costly, especially when cloud services charge per page.

pdf‑inspector Overview

pdf‑inspector is a pure‑Rust library (MIT‑licensed) that depends only on the lopdf parser. It has attracted over 11 k stars on GitHub.

PDF type classification

The library classifies PDFs into four categories: TextBased, Scanned, ImageBased, and Mixed. Classification samples the content stream for text operators without rendering pages, completing in 10–50 ms even for PDFs with 300+ pages. The result includes a confidence score (0–1) and a pages_needing_ocr list identifying pages without a text layer. In a cited example, a 150‑page report with 60 scanned pages required OCR for only those 60 pages.

Text extraction with positional data

Each extracted fragment retains font information and X/Y coordinates, enabling multi‑column layout detection. The library supports Type0/Identity‑H CJK encodings via ToUnicode CMap decoding and flags pages with broken encodings so downstream pipelines can fall back to OCR.

Structured Markdown output

Conversion to Markdown preserves heading hierarchy (H1–H4), ordered and unordered lists, tables, code blocks, bold/italic styling, and URLs (turned into links). It also removes page numbers, merges hyphenated line breaks, and compresses long dot‑leader table‑of‑contents entries.

Table recognition

Two strategies are employed: (1) analyzing PDF drawing commands to detect rectangular borders; (2) heuristic alignment of text cells when borders are absent. This handles numeric tables, multi‑page continuations, and footnotes, outputting native Markdown tables.

Benchmark (opendataloader‑bench, 200 PDFs)

pdf‑inspector: overall 0.875, reading order 0.915, table 0.814, title 0.788, time 0.47 s

liteparse: overall 0.873, reading order 0.913, table 0.693, title 0.811, time 0.75 s

opendataloader: overall 0.831, reading order 0.902, table 0.489, title 0.739, time 2.57 s

PyMuPDF4LLM: overall 0.735, reading order 0.886, table 0.401, title 0.424, time 17.12 s

MarkItDown: overall 0.589, reading order 0.844, table 0.273, title 0.000, time 16.17 s

pdf‑inspector leads in overall score, reading‑order accuracy, table detection, and speed (≈30× faster than the slowest competitor).

Language bindings and WebAssembly

Install via cargo install pdf-inspector (Rust), pip install pdf-inspector (Python), or npm install @firecrawl/pdf-inspector (Node). A WebAssembly build runs the same parser in the browser, keeping PDF bytes local.

Getting started

CLI usage:

cargo install pdf-inspector
pdf2md document.pdf               # convert to Markdown
detect-pdf document.pdf --json     # only classification
pdf2md document.pdf --select-pages 1,3,5-10

Python example:

import pdf_inspector
result = pdf_inspector.process_pdf("document.pdf")
print(result.pdf_type)   # "text_based", "scanned", "image_based", "mixed"
print(result.markdown)

Node.js example:

import { readFileSync } from "fs"
import { processPdf } from "@firecrawl/pdf-inspector"
const result = processPdf(readFileSync("document.pdf"))
console.log(result.pdfType)
console.log(result.markdown)

Typical use cases

Batch PDF pipelines: classify first, extract locally for text‑based pages, and send only scanned pages to OCR, reducing cost and latency.

RAG systems: structured Markdown improves chunking and model comprehension.

Browser‑based tools: the WebAssembly version processes PDFs entirely client‑side.

Limitations

Heading recognition score is slightly lower than liteparse (0.788 vs 0.811). The library provides no built‑in OCR for pure scans; an external OCR solution is required. Edge‑case layouts may fail more often than mature, long‑standing tools.

Repository

https://github.com/firecrawl/pdf-inspector