How to Effectively Address Reviewer Concerns About Small Experimental Gains

Problem Statement

Reviewers often comment that a paper’s improvement over the baseline is limited and lacks practical significance, asking whether the gain is statistically meaningful, useful in challenging settings, or brings other advantages.

❌ Undesirable Response

Many authors reply with a brief statement like “although the improvement is small, our method is still better than the baseline.” This repeats the result without explaining why the modest gain matters.

Is the improvement statistically significant?

Does it become larger in more difficult scenarios?

Does it reduce cost, improve stability, or enhance generalization?

Is the gain meaningful on key metrics?

Does it demonstrate a real contribution?

✅ Recommended Response Structure

Thank you for the observation. We agree that the magnitude of improvement should be interpreted in the context of task difficulty and real‑world relevance. Our contributions can be summarized in three points: 1. On the primary metric, our method improves by XXX over the strongest baseline and the gain passes statistical significance tests. 2. In more challenging settings (e.g., XXX ), the improvement is even larger, indicating stronger performance under difficult conditions. 3. The method achieves these gains while maintaining low computational cost, better stability, or stronger generalization, so the contribution is not limited to a single metric increase. We will add detailed statistical significance results, variance analyses, and explanations of performance differences across scenarios in the revised manuscript.

🎯 Core Answer Principles

Instead of repeatedly saying “our method is higher,” explain why the improvement is valuable.

1. Demonstrate Statistical Significance

We repeated the experiments with five different random seeds; the results are stable and pass statistical significance tests.

2. Highlight Gains in Difficult Scenarios

On hard samples, low‑resource settings, or cross‑domain configurations, our method shows a markedly larger improvement over the baseline.

3. Emphasize Additional Advantages

Our approach achieves the performance boost while keeping computational overhead low, offering better stability and suitability for deployment.

4. Contextualize the Task’s Improvement Ceiling

When the current strong baseline is already near saturation, even a modest, stable gain reflects meaningful progress.

One‑Sentence Summary

Small improvements should be justified by task difficulty, stability, cost, robustness, and statistical significance rather than merely stating “we are higher.”

GitHub repository: https://github.com/MLNLP-World/Paper-Rebuttal-Tips