Paper Generator Detects 92% Fake Conclusions, Automates Experiments and Figures

Overview

Spark‑to‑Paper is an end‑to‑end research‑paper generation system that leverages 13 composable “skills” inside a standard programming‑assistant environment, eliminating the need for a separate autonomous‑agent platform or external orchestration service.

Full Workflow

Starting from a brief research idea, the system expands the idea into a structured proposal, then sequentially performs planning, citation gathering, manuscript drafting, evidence‑driven revision, adversarial review, figure generation, and final assembly. The output is a complete LaTeX project containing manuscript, bibliography, figures, and template configuration that compiles without errors.

Automated Experimentation

During the planning stage the dataset, baseline, metrics, ablation plan, and table schemas are fixed in a lightweight pre‑registration. After the draft is written, the system back‑propagates missing evidence, runs a minimal set of experiments when code and data are available, records logs and metric files, and automatically fills the manuscript and redraws figures. Result figures are generated directly from measured data as vector PDFs, while method and schematic figures are first sketched by an image model and then rebuilt into editable HTML/CSS by the coding agent before being exported as PDFs.

Core Techniques

The pipeline is modular: each skill defines a goal, constraints, available tools, and deliverable standards, while a lightweight orchestrator determines task order. Two execution modes exist—proposal mode (no measurements, placeholders left blank) and data‑aware mode (quantitative conclusions must be backed by results). Deterministic gates enforce blueprint consistency, citation integrity, LaTeX compilation success, and placeholder rules. A two‑level self‑review mechanism checks semantic drift after each edit, and an adversarial review loop validates theoretical correctness, experimental design, and system effectiveness, only accepting revisions that survive multiple rounds. The system caps the experiment‑review‑revision cycle at seven iterations; exceeding this limit produces a failure report rather than a fabricated paper.

Evaluation

On eight controlled research topics Spark‑to‑Paper achieved 99.5% citation validity (human pre‑prints 97.8%, AI Scientist 93%, AI Scientist‑v2 91%, Agent Laboratory 96%, single‑pass baseline 81%) and 96.4% editable‑figure rate (human 58%, AI Scientist and Agent Laboratory 0%). In a fake‑conclusion detection test with 36 injected false claims, detection rose from 14% (single pass) to 69% (gate‑only), 81% (gate + self‑review), and 92% (full stack). Adversarial‑review precision reached 74%. The full system consumes on average 11.9 M tokens, costs $8.1, and takes 3.2 hours per paper, whereas the single‑pass baseline uses 0.11 M tokens, $0.66, and 16 minutes but yields lower citation and evidence quality.

Open Source and Community

The code, datasets, and resources are released under an open‑source license on GitHub (https://github.com/Spark-To-Paper-Skills/spark-to-paper-skills), with a project website, the accompanying arXiv paper (https://arxiv.org/abs/2608.11924), and a Hugging Face entry (https://huggingface.co/papers/2608.11924). The authors invite developers, researchers, and domain experts to contribute feedback or extensions.