How the Open‑Source “book‑to‑skill” Tool Eliminates PDF‑AI Hallucinations and Cuts Token Costs

Problem

Reading large technical PDFs with generative AI often exceeds the model’s context window, incurs high token costs, and can produce hallucinated answers. Two mainstream approaches are commonly used:

Direct PDF upload : loads the entire book at query time, leading to extreme token consumption and high hallucination risk.

Retrieval‑Augmented Generation (RAG) : slices the document, retrieves matching passages, and concatenates them, which limits understanding to keyword matches and can yield fragmented answers.

book‑to‑skill Overview

book‑to‑skill is an MIT‑licensed open‑source utility that compiles technical books, office documents, and internal manuals into a structured Skill format compatible with Claude Code, Open Code, Codex and similar AI coding assistants. The tool parses the whole book during a compilation phase, preserving tables, code blocks, headings, and technical patterns, and outputs five standardized files:

SKILL.md – overall mental model and chapter index (≈ 4000 tokens for a 400‑page book)

Chapter files – each chapter summarized to 800‑1200 tokens (≈ 1000 tokens per chapter)

glossary.md – alphabetically sorted term list with chapter references (≈ 1500 tokens)

patterns.md – collected algorithms, design patterns, and practical solutions (≈ 2000 tokens)

cheatsheet.md – quick‑reference table of core rules and anti‑patterns (≈ 1000 tokens)

The compilation produces a compact skill package of roughly 20 000 tokens for a typical 400‑page technical book, which can be loaded on demand, drastically reducing token usage.

Supported Formats

book‑to‑skill natively handles nine document types without conversion:

PDF, EPUB, MOBI/AZW/AZW3

DOCX, TXT

Markdown, reStructuredText, AsciiDoc, HTML, RTF

Scanned PDFs, DRM‑protected files, and image‑only documents cannot be parsed and must be OCR‑converted first.

Performance Comparison

Three methods were evaluated on an official 103‑page benchmark across five dimensions (working timing, core ability, hallucination probability, token consumption, suitable scenario):

Direct PDF upload : loads full text at query time, incurs extremely high token cost, and has a very high hallucination probability; suitable only for quick keyword look‑ups.

RAG : performs slice‑retrieve‑concatenate at query time, incurs medium token cost, and has a moderate hallucination risk; suitable for batch retrieval across many books.

book‑to‑skill : compiles once, then loads only the core framework plus requested chapters; token cost is minimal and hallucination probability is near zero; ideal for deep, repeated reference of a single book.

Extraction Speed Test

Two extraction modes were measured on the same 103‑page technical PDF: pdftotext (pure‑text mode): 0.1 s, but loses all tables and code blocks. Docling (layout‑aware Technical mode): 164 s, retains 48 tables and 36 code blocks, making it the preferred choice for technical books despite the longer runtime.

Compilation Workflow

Classify the document as technical or narrative and select the appropriate parser (pdftotext for narrative, Docling for technical).

Extract raw text.

If the book exceeds ~50 K tokens (≈ 130 pages), split it by chapter to avoid loading the entire text at once.

Analyze structure to identify titles, sections, and the author’s framework.

Summarize each chapter to 800‑1200 tokens while preserving code snippets and tables for technical sections.

Generate the five skill files and place them under ~/.claude/skills/, with SKILL.md at the top for optimal AI context compression.

Apply design principles: prioritize density over completeness, produce scenario‑driven output, and store only extracted summaries to respect copyright.

Installation & Usage

Manual installation requires creating the skill directory and downloading the core files:

mkdir -p ~/.claude/skills/book-to-skill/scripts
curl -o ~/.claude/skills/book-to-skill/SKILL.md https://raw.githubusercontent.com/virgiliojr94/book-to-skill/master/SKILL.md
curl -o ~/.claude/skills/book-to-skill/scripts/extract.py https://raw.githubusercontent.com/virgiliojr94/book-to-skill/master/scripts/extract.py

After installation, compile a document with the /book-to-skill command (optionally providing a custom skill name). Queries can then be made using: /skill-name – loads the full book framework. /skill-name keyword – searches for a specific knowledge point. /skill-name ch05 – jumps directly to chapter 5.

Common Pitfalls & Mitigations

Books lacking clear chapter markers may be treated as a single chapter; enable Docling’s Technical mode for such cases.

Scanned or DRM‑protected PDFs must be OCR‑converted before processing.

Docling is CPU‑intensive (~1.5 s per page); a 103‑page book takes about 3 minutes.

Low‑end machines (8 GB RAM) should avoid parallel processing of multiple books.

External‑skill permissions must be enabled in Open Code, or the skill directory must be installed before launching Claude Code.

Recommended Scenarios

Frequent reference of classic technical books (e.g., DDIA, Clean Code, Python manuals).

Company‑internal documentation such as architecture guides, operations manuals, and onboarding material.

New or foreign technical publications not covered by the AI’s pre‑training data.

Personal note collections and fragmented resources that need unified management.

When Not to Use

Massive cross‑book retrieval; RAG or NotebookLM is more efficient.

Books already well‑represented in the AI’s knowledge base; compilation yields little benefit.

Pure image‑only, scanned, or encrypted PDFs; extra OCR/decryption steps make the workflow cumbersome.

Resources

GitHub repository: https://github.com/virgiliojr94/book-to-skill