AI Reviewers Fooled by Rewrites? 1,200 Papers Tested Reveal SciCore Dual-Branch Fix

Experimental Setup

Researchers from Virginia Tech, University of Maryland, and MBZUAI selected 60 ICLR 2026 submissions that matched arXiv source files, stratified by human average review scores into six bins, and randomly sampled 10 papers per bin. Each paper was rewritten directly on the full LaTeX project using GPT-5.5 and Claude Opus 4.8 under 10 distinct strategies: strengthening novelty framing, adjusting scope and result presentation, reorganizing contributions, changing technical language and sentence complexity, multi-dimensional joint rewriting, two- or three-round continuous rewriting, and revision guided by reviewer feedback. Each model produced 20 rewrites per paper (10 strategies × 2 models), yielding 1,200 rewritten versions alongside the 60 originals. Rewrites preserved methods, experimental numbers, and conclusions, and passed compilation checks and content-fidelity audits.

RobustReview Benchmark and Metrics

The team evaluated 30 AI reviewer configurations: eight general LLMs each with three reviewing strategies (standard, strict, continuous emphasis on scientific content), three specialized reviewer models, and three agent-based review systems retaining their native pipelines. To avoid rewarding trivial stability (e.g., giving every paper a 6), they defined rhetorical robustness as two joint requirements: (1) scores for the same paper across rewrites should be stable (low MAD, low score-drift std, high ICC, high Signal Preservation Ratio SPR), and (2) the reviewer must still discriminate between different papers (high paper discrimination ability). Agreement with human average scores was assessed separately via absolute error and Spearman rank correlation.

Key Finding: False Robustness

Gemini 3.5 Flash-Lite with standard reviewing achieved the lowest MAD (0.205) among all 30 configurations, suggesting high stability. However, its paper discrimination ability was only 0.511—near the 0.5 random baseline—and ICC was merely 0.199. The authors term this false robustness : low drift coexists with an inability to tell papers apart because the model assigns nearly identical scores to all papers. In contrast, GPT-5.5 under continuous emphasis on scientific content achieved MAD 0.476, ICC 0.697, and discrimination 0.682, leading on all three joint robustness metrics. Prompting alone did not guarantee improvement: Claude Sonnet 5 with continuous emphasis reduced MAD from 0.468 to 0.307 but also dropped ICC from 0.579 to 0.489 and discrimination from 0.646 to 0.615—stability improved yet joint robustness declined. GPT-5.5's strict reviewing aligned better with human ranking, while continuous emphasis yielded better joint robustness.

SciCore Dual-Branch Method

To reduce phrasing sensitivity, the authors introduced SciCore , a dual-branch architecture:

Branch 1 (Full Paper): GPT-5.5 reviews the complete PDF under strict rules.

Branch 2 (Science Core): A structured scientific record is first extracted—covering research question, method and hypotheses, experimental results, reproducibility information, and author-stated limitations. Extraction preserves key formulas and exact numbers, fully transcribes tables, marks unreadable image content, and does not pre-judge quality. The same GPT-5.5 then reviews this Science Core under a tailored strategy. The final score is an equal-weight average: Score = 0.5 * M(x) + 0.5 * B(x).

Science Core Extraction Stability

Using GPT-5.5 as extractor, the average cosine similarity between Science Cores of the same paper across rewrites was 0.978 , while similarity between different papers' Science Cores was only ~0.62 . This confirms that the extraction normalizes rhetorical variation while preserving inter-paper differences.

SciCore Fusion Results

Compared to GPT-5.5 strict review of the full paper alone, SciCore reduced MAD from 0.766 to 0.476 (38% reduction), raised ICC to 0.775 , discrimination to 0.726 , and SPR to 0.652 —leading on all three joint robustness metrics in the primary comparison. Absolute error vs. human average was 1.072 (vs. 1.078 for full-paper strict), and Spearman correlation was 0.488 (vs. 0.529).

Ablation and Fusion Weight Analysis

Reconstructing a full paper from the Science Core and then reviewing it under standard rules underperformed direct Science Core review on all seven metrics. Varying the fusion weight showed that increasing the Science Core share improves rhetorical robustness but reduces agreement with human scores; the 50/50 split balanced both.

Control: Score Averaging vs. SciCore

Averaging two full-paper review strategies (strict + continuous emphasis) gave ICC 0.758 and discrimination 0.715; adding a third strategy (standard) raised ICC to 0.781 but discrimination remained 0.715—still below SciCore's 0.726. Thus, SciCore's gain is not merely from averaging.

Repeated Review and Bootstrap Validation

Repeated review of the same PDF yielded similar intrinsic MAD (SciCore 0.320 vs. strict 0.329). Across rewrites, SciCore MAD was 0.476 vs. strict 0.766. A 5,000-iteration paired bootstrap resampling by paper supported SciCore's improvement on all five robustness metrics.

Cross-Model Generalization

The Science Core branch was tested with GPT-5-mini and GLM-5.2 as the reviewer backbone. Compared to each model's standard full-paper review, GPT-5-mini improved MAD, drift std, and ICC but saw SPR and discrimination drop; GLM-5.2 improved all seven metrics. Full dual-branch SciCore transfer to other backbones remains to be validated.

Conclusion

The study demonstrates that no writing style guarantees high scores from AI reviewers. Rhetorical variation can sway some AI reviewers, and prompt-based robustness often collapses into false robustness. RobustReview provides a benchmark for evaluating rhetorical robustness, and SciCore's dual-branch design—grounding one branch in a normalized Science Core—substantially improves stability while maintaining discrimination and human alignment. Further validation across more models and disciplines is needed.