AI Review Pilot at AAAI-26: 22,977 Papers Processed in 24 Hours, Accuracy Outperforms Human Reviewers

Background and Motivation

Scientific publishing is experiencing an exponential growth in submission volume: AAAI‑26 received over 30,000 initial papers, roughly double the previous year, while the reviewer pool grew only threefold, leading to a severe scale‑crisis in peer review.

Prior work on AI‑assisted reviewing has been limited to post‑hoc analyses, narrow use‑cases such as author‑checklists, or small‑scale experiments that never deployed AI‑generated reviews for an entire conference.

Core Question: Can a state‑of‑the‑art AI system produce technically meaningful, practically useful reviews at the scale of a top‑tier conference?

System Design and Architecture

Overall Workflow

The pipeline treats the PDF of each submission as the sole input, converts it to a Markdown representation that preserves LaTeX formulas and tables, and then runs the paper through five specialized review stages (Story, Presentation, Evaluations, Correctness, Significance) before a final synthesis step.

Key components:

Pre‑processing : PDFs are resampled to 250 DPI to limit image token consumption, then processed with olmOCR to produce a LaTeX‑aware Markdown file.

LLM Choice : OpenAI GPT‑5 (high‑reasoning mode) provides a 400 k token context window and a zero‑data‑retention policy to protect double‑blind anonymity.

Prompt Hierarchy : Nine prompt modules (basic instruction, Story, Presentation, Evaluations, Correctness, Significance, Initial Review, Self‑Critique, Final Review) are chained sequentially, each optionally invoking tools such as code_interpreter or web_search.

def preprocess_pdf(pdf_path, target_dpi=250):
    # Convert PDF pages to images at target DPI
    images = convert_from_path(pdf_path, dpi=target_dpi)
    return images

def convert_to_markdown(pdf_path):
    """Use olmOCR to extract LaTeX‑rich Markdown from a PDF."""
    markdown = olmocr_convert(pdf_path)
    return markdown

Multi‑Stage Review Pseudocode

SPECS_STAGES = ["story", "presentation", "evaluations", "correctness", "significance"]
TOOLS = {
    "evaluations": ["code_interpreter"],
    "correctness": ["code_interpreter"],
    "significance": ["web_search_preview"]
}

def generate_review(pdf_path, markdown_path):
    context = {"pdf": pdf_path, "markdown": markdown_path, "base_instruction": BASE_INSTRUCTION, "stage_outputs": {}}
    for stage in SPECS_STAGES:
        tools = TOOLS.get(stage, [])
        stage_prompt = load_stage_prompt(stage)
        output = call_gpt5(context=context, stage_prompt=stage_prompt, tools=tools,
                         reasoning_effort="high", max_retries=5, backoff="exponential")
        context["stage_outputs"][stage] = output
    initial_review = generate_initial_review(context)
    critique = self_critique(initial_review, context["pdf"])
    final_review = revise_with_critique(initial_review, critique)
    return final_review

SPECS Benchmark and Experimental Results

To evaluate the system, the authors built a synthetic benchmark (SPECS) that injects five types of scientific errors into 120 real papers (783 perturbed instances). Human annotators confirmed that ~63 % of the injected errors were detectable.

When the full pipeline was applied, detection rates improved significantly over a single‑prompt baseline across all five dimensions. For example, the Story dimension rose from 0.353 (baseline) to 0.673 (full system), and the overall SPECS score increased from 0.429 to 0.639, a statistically significant gain (McNemar test, p < 0.01).

Key Finding: The multi‑stage framework consistently outperforms a monolithic prompt, providing the strongest statistical evidence to date that staged AI review is superior.

Large‑Scale User Survey

A questionnaire was sent to all authors, program committee members, senior program committee members, and area chairs (5,834 valid responses). Participants rated AI‑generated reviews on nine quality criteria using a five‑point Likert scale.

AI reviews scored higher than human reviews on 6 of 9 criteria, with the largest advantages in technical error detection (+0.67) and surfacing previously unconsidered viewpoints (+0.61).

53.9 % of respondents found the AI reviews useful for the current submission, and 61.5 % believed they would be useful in future reviews.

Only 13.8 % of reviewers reported that AI feedback changed their final decision, indicating a complementary rather than substitutive role.

Limitations and Future Directions

Despite strong performance, the system has notable weaknesses:

Weakness in assessing overall significance and novelty.

Occasional mis‑reading of complex equations and tables.

Over‑verbosity leading to cognitive overload.

Potential self‑selection bias in the survey sample.

Knowledge cutoff (Sept 2024) limits up‑to‑date literature checks.

Future research avenues include optimal human‑AI division of labor, long‑term effects of AI assistance on reviewer behavior, defenses against adversarial paper tailoring, cross‑disciplinary applicability, continual benchmark updates to avoid data contamination, and economic analyses as inference costs continue to drop.

Conclusion

The AAAI‑26 AI Review Pilot demonstrates that AI‑assisted peer review is operationally feasible at conference scale (22,977 papers, < $1 per paper, 24 h turnaround) and can deliver higher perceived quality than human reviewers on several dimensions, while still requiring human oversight for high‑level judgment.