How Large Language Models Are Revolutionizing Automated Scholarly Paper Review

1. Introduction

The authors highlight a recent incident where a research team embedded a hidden instruction in a paper to manipulate AI‑based peer review, illustrating that automated scholarly paper review (ASPR) is already in practice. In response, a team from Guangzhou University of Technology published a comprehensive survey titled Large Language Models for Automated Scholarly Paper Review: A Survey to map the state‑of‑the‑art.

2. Model Landscape in ASPR

Table 1 (visualized in Figure 1) summarizes the large models used for ASPR. Open‑source models such as Llama 3, Mistral, and Qwen 2 dominate in terms of diversity, accounting for roughly 2.7 times the number of closed‑source models. Closed‑source models (GPT‑4, Gemini 1.5, Claude 3) lead performance on complex tasks like defect detection but raise transparency and auditability concerns.

3. Technical Advances Driven by Large Models

Long‑text modeling: Models like gpt‑3.5‑turbo‑16k (June 2023) expanded token limits, enabling full‑paper processing and richer context understanding.

Multimodal input: New multimodal models can ingest tables, figures, and other non‑textual elements, improving review completeness.

Multi‑turn dialogue: Advances now allow models to simulate the iterative exchange between authors and reviewers, supporting continuous feedback.

Real‑time knowledge retrieval: Integrated search capabilities let models fetch up‑to‑date information during review, enhancing factual accuracy.

4. Generation Methods for Review Reports

Prompt engineering: Designing structured prompts and templates to guide models toward high‑quality, formatted review comments.

Supervised fine‑tuning: Leveraging labeled datasets (e.g., ReviewMT, LimGen) to adapt models to the specific language of peer review.

Multi‑agent frameworks: Systems such as AgentReview, SEA, MARG, and MAMORX coordinate several model agents to emulate authors, reviewers, and editors, improving consistency and multimodal handling.

5. Emerging Datasets

Table 2 lists recent datasets covering stages from initial screening to reviewer feedback and quality assessment. Some datasets provide multi‑turn dialogue logs, enabling models to learn iterative review dynamics, while others include expert‑annotated quality labels.

6. Open‑Source Codebases

Table 3 aggregates publicly released ASPR code. Notably, the 2024 open‑source system MAMORX integrates multiple modules to mimic human review pipelines and outperforms baseline NLP models and even human reviewers on benchmark tasks.

7. Benefits and Performance Gains

Efficiency: Large models quickly assess manuscript suitability and identify issues, reducing reviewer workload while maintaining comparable acceptance rates.

Abstract generation: Extractive summarization yields fluent, fact‑checked abstracts that surpass human‑written references on fluency and factuality.

Screening assistance: Models aid editors in plagiarism detection, topic relevance, and pre‑print triage, achieving up to 90 % accuracy on abstract screening.

Checklist verification: Current systems verify compliance with ethical and formatting standards with an 86.6 % accuracy comparable to humans.

Error detection: Models can spot mathematical and conceptual mistakes, approaching expert performance.

Opinion optimization: By leveraging large corpora of review reports, models can rewrite reviewer comments into polished, insightful feedback.

8. Current Limitations

The survey identifies several challenges: insufficient domain knowledge, bias toward moderate scores, hallucinations and factual errors, data security risks, limited customization for journal‑specific guidelines, and homogeneous review styles.

9. Publisher Policies

Most publishers prohibit reviewer use of AIGC tools due to confidentiality concerns; a minority allow internal AI assistance under strict controls. Table 4 (illustrated in the accompanying figure) summarizes these policies.

10. Recommendations for the Research Community

Ensure data security by deploying private, on‑premise models or restricting sensitive inputs.

Provide ethics and bias training for researchers integrating LLMs into review workflows.

Require transparent disclosure of AI assistance and watermarking of generated content.

Adopt the latest model versions and ensemble scoring to improve reliability.

Align model objectives with scholarly values and monitor compliance with journal guidelines.

11. Future Directions

Mitigate hallucinations through dynamic knowledge integration and multimodal fact‑checking.

Develop high‑quality multimodal datasets to advance visual‑text review capabilities.

Explore reasoning‑oriented models that emulate critical thinking while managing computational costs.

Defend against generative attacks via robust model training and system‑level security measures.

Enable low‑resource private model deployment through compression and cost‑effective scheduling.

Personalize review pipelines by modeling reviewer styles and adapting standards per discipline.

12. Conclusion

The survey demonstrates that large language models have evolved from simple formatting tools to sophisticated systems capable of handling long texts, multimodal inputs, and multi‑turn dialogues. Open‑source and closed‑source models each offer distinct advantages, and despite ethical and technical hurdles, ongoing dataset releases, codebases, and online platforms signal rapid progress. As ASPR matures, it promises to dramatically boost review efficiency and fairness, reshaping the scholarly publishing ecosystem.