Paper2Agent: Turning Papers into Agents – Rethinking Knowledge Bases

Introduction

Last week, while browsing ArXiv, the author encountered a project called Paper2Agent. Initially dismissed as another GPT wrapper, closer inspection revealed it can transform a PDF paper directly into an interactive agent that not only answers questions but also calls tools to run experiments and reproduce results. This prompts a reevaluation of whether the RAG knowledge bases many teams have built need a new approach.

What Paper2Agent Does

Paper2Agent's core pipeline takes a paper PDF through three stages: structured parsing, knowledge extraction, and tool binding, outputting an agent that can converse, reason, and invoke tools. Unlike traditional paper QA systems that rely on chunking, vectorization, retrieval, and prompt concatenation, Paper2Agent extracts methodology, experimental design, and data processing workflows into a structured knowledge graph and executable toolchains to equip an agent.

For example, given a graph neural network paper, it can not only answer what loss function was used but also translate the model architecture into a runnable code framework and help tune hyperparameters for ablation studies. Its tech stack uses mature 2026 components: long-document structured parsing with Claude/GPT, dynamic tool synthesis, and multi-step reasoning chains with self-verification.

Bottlenecks of Traditional RAG Knowledge Bases

Over the past two years, nearly every enterprise knowledge management team has adopted RAG (Retrieval-Augmented Generation). The author has deployed several projects and finds RAG solves about 60% of scenarios, but the remaining 40% of problems are increasingly clear:

Chunking granularity dilemma: Too fine loses context; too coarse reduces retrieval precision. An 8,000-word technical document split at 512 tokens often breaks a complete derivation. A user asking "why this algorithm?" may retrieve only the conclusion without the reasoning.

Vector retrieval semantic ceiling: Embedding models struggle to distinguish specialized terminology. In a pharmaceutical project, "pharmacokinetics" and "pharmacodynamics" showed cosine similarity of 0.93 across several mainstream embedding models, yet they are completely different concepts professionally.

Lack of reasoning ability: RAG is essentially retrieval plus concatenation; it does not reason. Asking "what is the methodological difference between paper A and paper B" forces RAG to retrieve both separately and let the LLM do textual comparison, unable to perform true methodological analysis.

Underestimated maintenance cost: Knowledge bases require ongoing upkeep. Document updates demand re-chunking and re-vectorization; embedding model upgrades (e.g., BGE-M3 to Arctic Embed 2) require full index rebuilds. A medium-sized knowledge base consumes half an FTE just for daily maintenance.

Technical Comparison: Retrieval-Augmented vs. Paper-as-Agent

The fundamental difference: RAG treats knowledge as "material"; Paper2Agent treats knowledge as "capability."

Illustrative scenario: User asks, "How does the attention optimization method proposed in this paper perform when sequence length exceeds 16K?"

RAG approach: Retrieves text chunks containing "attention" and "sequence length" keywords, concatenates into prompt for LLM answer. If the paper lacks explicit 16K experimental data, the answer is essentially "not mentioned."

Paper2Agent approach: Understands the methodology, automatically assembles an evaluation script, configures 16K-length test data, runs it, and returns actual results.

This gap cannot be bridged by "better retrieval"; it is a paradigm-level difference.

Should You Rebuild Your Knowledge Base? Three Evolution Strategies

The author advises against a full teardown; instead, adopt layered evolution based on business scenarios:

Strategy 1: RAG Baseline + Agent Enhancement (Recommended for Most Teams)

Keep existing RAG as the basic information retrieval layer. On top, build an agent layer for high-value documents (core papers, key technical designs). A routing mechanism coordinates both: simple factual queries go to RAG; reasoning and tool-calling tasks go to Agent. Pros: controlled investment, low risk. Cons: maintenance costs of two systems stack.

Strategy 2: Full Shift to Agent Architecture (Suitable for New Projects)

If no knowledge base exists yet or the current one performs poorly, go straight to agent architecture. As of 2026, agent frameworks are mature: LangGraph, CrewAI, Anthropic's Claude Agent SDK are ready to use. Tool ecosystem is far better than two years ago, no need to reinvent wheels. However, note cost: agent inference consumes 3-5x more tokens than RAG, and long-chain reasoning adds latency. High-concurrency, low-latency customer service scenarios cannot yet be handled by agent architectures.

Strategy 3: Hybrid Knowledge Graph + Agent (For Heavy Knowledge Management)

Teams managing thousands of papers and technical documents (e.g., pharma R&D) can consider structuring with a knowledge graph first, then mounting agent capabilities on graph nodes. GraphRAG gained traction in late 2025, using knowledge graphs to补足 pure vector retrieval's relational reasoning gaps. Paper2Agent's knowledge extraction module naturally outputs graph structures, making integration straightforward.

Practical Advice and Lessons Learned

Key operational insights from project experience:

1. PDF parsing is a major hurdle. Paper2Agent's effectiveness heavily depends on front-end document parsing quality. Formulas, tables, and image captions in papers are poorly extracted by traditional PyPDF2. The best-tested combination: Marker or MinerU for layout analysis, followed by LLM semantic completion. This single step can improve final QA accuracy by ~15 percentage points.

2. Tool generation requires sandboxing. Auto-generated code from Paper2Agent does not always run successfully. The solution: assign each agent an isolated Docker container as execution environment, with gVisor for additional security isolation. Never run directly on the host — the author witnessed an agent generate a rm -rf operation, unintentional but severe.

3. Evaluation metrics need redesign. Traditional RAG metrics (Faithfulness, Answer Relevancy) are insufficient for agents. Agents not only generate text but execute actions; you must evaluate tool-calling accuracy, multi-step reasoning logical consistency, and execution result correctness. The author's team built a trajectory-based evaluation framework scoring: step rationality, tool selection accuracy, final result correctness.

4. Cost control is a hard constraint. Processing a 20-page paper end-to-end (parsing + modeling + tool binding) consumes ~300k-500k tokens. At current Claude Sonnet 5 pricing, that's ~$0.6-1 per paper. For thousands of papers, cost is significant. A compromise: use a lightweight model (e.g., Haiku 4.5) for initial screening and structuring, reserving large models for deep agentification of high-relevance papers only.

5. Don't rush full migration. Pick 3-5 core papers to run the full pipeline, gather user feedback, verify agent output quality and stability. The author's first pilot used the 5 most-cited methodology papers; after validation, scope expanded gradually. The whole stabilization took about 6 weeks.

Conclusion

Paper2Agent represents not just a tool but a paradigm shift in knowledge management — from "store-retrieve" to "understand-execute." This does not mean discarding RAG. Technology selection always balances cost, effectiveness, and complexity.

For most teams, the recommendation: don't rebuild the knowledge base, but start introducing agent capabilities at critical nodes. Run both paradigms in parallel for a while, let data decide the future architecture direction. Ultimately, there is no right or wrong technical solution, only what fits your current business stage and team capability.