Skip to content
Google's PaperOrchestra: 5-Agent Pipeline Automates Research Paper Writing in 40 Minutes

Google's PaperOrchestra: 5-Agent Pipeline Automates Research Paper Writing in 40 Minutes

Google's PaperOrchestra is an open-source multi-agent pipeline that converts research ideas and experiment logs into submission-ready LaTeX papers using five specialized agents, featuring automated rollback on quality regression, Semantic Scholar-verified citations, and a benchmark suite, completing a paper in ~40 minutes with 60-70 model calls.

Architecture Digest
Architecture Digest
Architecture Digest

Introduction

Google Cloud AI Research has open-sourced PaperOrchestra, a multi-agent pipeline that transforms research ideas and experiment logs into conference-ready LaTeX manuscripts. The system addresses a common pain point: researchers finish experiments but struggle to write papers efficiently.

Project Overview

PaperOrchestra is not a chatbot but a production line. It requires four inputs: a research idea description, experimental logs, a conference LaTeX template (CVPR 2025 or ICLR 2025 provided), and submission guidelines. The pipeline consists of five specialized agents that pass file-based intermediate artifacts: outline.json, references.bib, citation mappings, and LaTeX drafts. Each step produces inspectable files rather than relying on opaque agent conversations.

PaperOrchestra pipeline overview
PaperOrchestra pipeline overview

Cost analysis: the code is free but model calls are not. Default model is gemini-3.1-pro-preview, requiring Gemini or Vertex credentials plus a Semantic Scholar API key for literature retrieval. A full run averages 60–70 model calls and 39.6 minutes per paper.

Three Key Highlights

Highlight 1: Rollback Mechanism — Quality Gate with Compilation and Scoring

The Content Refinement Agent compiles the LaTeX to PDF, scores it on seven dimensions (originality, quality, clarity, significance, reliability, expression, contribution) each rated 1–4, aggregated to a 1–10 total. A new version is accepted only if the total score increases; if it decreases, the system rolls back to the previous version and stops. If tied, it compares per-dimension gains and losses. After content refinement, a visual check renders PDF pages as images for a vision model to detect figure overflow and layout issues.

Rollback mechanism diagram
Rollback mechanism diagram

This design turns "polish again" into an engineered process with compilation and scoring gates, preventing worse versions from silently overwriting better ones. Limitation: the reviewer and author share the same model class, so they may share biases; it serves as a quality gate, not a substitute for human sign-off.

Highlight 2: Separated Literature Retrieval and Verification

The Literature Review Agent decomposes retrieval tasks from the outline, then uses Semantic Scholar API to fetch titles, authors, years, abstracts, and citation counts. Papers without abstracts or published after the research cutoff are filtered out. The agent outputs BibTeX and citation mappings. The architecture explicitly separates web search (identify) from Semantic Scholar API (verify). API verification confirms a paper's existence, making citations more reliable than pure model memory.

Literature retrieval and verification flow
Literature retrieval and verification flow

Reported results on the ScholarPeer simulated acceptance benchmark: PaperOrchestra achieves 84% (CVPR) and 81% (ICLR), compared to AI Scientist-v2 at 70% and 64%, and human originals at 86% and 94%. Caveat: API only guarantees paper existence, not that it supports the specific claim; manual verification remains necessary.

Highlight 3: PaperWritingBench and Autoraters — The Real Main Contribution

The authors released PaperWritingBench: 200 accepted top-tier papers (100 CVPR 2025, 100 ICLR 2025) reverse-engineered into pre-writing materials — sparse ideas, dense ideas, and experimental logs — with narratives stripped and anonymized. The repository includes four autoraters: literature review quality, citation F1, simulated peer review, and side-by-side comparison. The paper reports a 0.6458 correlation between GPT-5 autorater scores and human preferences. This benchmark and evaluation suite can be used independently to assess any paper-writing agent.

PaperWritingBench and autoraters
PaperWritingBench and autoraters

Potential Issues and Limitations

Limitations summary
Limitations summary

Scope boundary: It only handles writing, not research. Users must provide genuine ideas and logs; it does not generate hypotheses, run experiments, or fabricate data.

Human authors still win: Official human evaluation on 40 papers (180 paired comparisons) shows PaperOrchestra beats AI baselines on literature review (67% and 73% win rates) but achieves only 5% overall win rate against human originals (humans win 81%). Literature review vs humans: 19% vs 57%. The abstract's claimed 50–68% win rate is against AI baselines only.

Plotting dependency on PaperBanana: The framework diagram shows Plotting's backbone as PaperBanana, and the code references paper_banana_utils.py pointing to ../../PaperBanana. This external repository (with its reference datasets and style guides) is not included; users must clone PaperBanana separately and place it correctly. README does not document this.

Cost by minutes and calls: 60–70 calls and 39.6 minutes per paper versus a single-prompt baseline of 1 call and 3.3 minutes. The division of labor, coverage, and rollback mechanism trade call count for quality. Running 100 benchmark papers consumes thousands of calls.

Engineering maturity is research code, not product: 8 commits, 2 contributors, 43 PRs with only 3 merged, 15 open issues. Recent fixes (Nov 15) addressed LaTeX prompt escaping, completed requirements.txt, and straightened model-key logic (previously wrong keys caused immediate startup failure). Only CVPR 2025 and ICLR 2025 templates are included; other conferences require custom templates following the templates/ structure.

Installation and Configuration

Prerequisites: conda, Python 3.11; Gemini API key or Vertex project; Semantic Scholar API key; local pdflatex for PDF compilation during refinement; PaperBanana repo and data if plotting is enabled.

git clone https://github.com/google-research/paper-orchestra
conda create -n paper_orchestra python=3.11 && conda activate paper_orchestra
pip install -r requirements.txt

Create .env in root with GEMINI_API_KEY and SEMANTIC_SCHOLAR_API_KEY Prepare raw_materials/ (at least idea.md and experimental_log.md), then run:

python paper_writing_cli.py --raw_materials_dir ... --latex_template_dir templates/iclr2025 --use_plotting true

Key CLI options: --writer_model_name (default gemini-3.1-pro-preview; change for OpenAI and set OPENAI_API_KEY), --research_cutoff (default current year-month, controls literature cutoff), --use_plotting (default false).

Verification: output_dir should contain template.tex, references.bib, and compiled PDF. Missing bibliography indicates Semantic Scholar key not working.

Two practical pitfalls: (1) Without plotting enabled and no figures provided, startup warns about missing images and LaTeX compilation may fail — either enable --use_plotting true or pre-populate a figures/ directory. (2) Streamlit interface may leave orphan background processes; official workaround: pkill -f paper_writing_cli.py. Environment setup takes ~30 minutes if smooth; first paper adds ~40 minutes.

Personal Assessment

The deciding factor: are you outsourcing the writing or the research?

For researchers with completed experiments but stuck on writing, this is the most structured solution available: file-based intermediates allow section-by-section inspection; rollback prevents quality regression; citations are API-verified. The included autoraters and PaperWritingBench are valuable alone for evaluating any paper-writing agent.

Do not use if you expect it to do research. Its 5% win rate against humans confirms it solves packaging existing research into a manuscript, not conducting research.

Two groups should skip: those not targeting CVPR/ICLR-style conferences (template customization drops ROI), and those without a LaTeX environment unwilling to install one (refinement step requires pdflatex).

GitHub repository: https://github.com/google-research/paper-orchestra
Original Source

Signed-in readers can open the original source through BestHub's protected redirect.

Sign in to view source
Republication Notice

This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactadmin@besthub.devand we will review it promptly.

Benchmarkmulti-agentAI for scienceGoogle ResearchSemantic ScholarPaperOrchestraLaTeX generationresearch paper writing
Reader feedback

How this landed with the community

Sign in to like

Rate this article

Was this worth your time?

Sign in to rate
Discussion

0 Comments

Thoughtful readers leave field notes, pushback, and hard-won operational detail here.