Responding to Reviewers Who Request Large‑Scale Experiments Unfeasible in Rebuttal

Background

The MLNLP community released the Paper‑Rebuttal‑Tips project to help authors address reviewer requests for larger‑scale experiments that cannot be completed within the rebuttal window.

Problem

Reviewers often suggest validating on larger datasets, models, or settings, noting that “complete experiments are difficult to finish within the rebuttal cycle.”

These requests are reasonable, but limited time, compute resources, and experiment cycles may prevent full validation.

What Not to Say

Many authors reply with a brief statement such as:

“The experiment cannot be performed during the rebuttal phase.”

This answer fails to address the reviewer’s underlying concerns:

Will larger experiments change the paper’s conclusions?

Are the current results representative?

Does the method scale to larger settings?

Can any indirect experiments partially verify the claim?

Recommended Reply Structure

A stronger response should:

Explain the cost of the full experiment.

Provide a representative smaller experiment that directly tackles the reviewer’s core question.

Interpret the supplemental results and their relevance to the paper’s claims.

Example response:

Thank you for the suggestion. We agree that larger‑scale validation would further strengthen the evidence. However, a complete experiment would require roughly XXX GPU‑hours on XX GPUs for XX days , making it infeasible within the rebuttal period. To address your concern, we added a small‑scale experiment that directly tests XXX . Although limited in size, it confirms that the method still yields a stable improvement, indicating potential scalability. We will include full large‑scale results in the final version.

Core Answer Details

1. Explain Full‑Experiment Cost

Specify concrete resources rather than a vague “limited compute” claim. Mention GPU hours, number of GPUs, runtime, data scale, and algorithmic complexity. Example: “The full experiment needs about XXX GPU‑hours, which cannot be finished in the rebuttal window.”

2. Provide a Representative Experiment

If the full experiment is impossible, choose one of the following smaller alternatives:

Use a reduced dataset.

Employ a smaller model.

Limit the number of experimental settings.

Run a focused analysis experiment.

Clearly state that, despite its limited scale, the experiment directly addresses the reviewer’s question.

3. Explain the Significance of the Supplemental Result

Do not just present the outcome; discuss whether it supports the original conclusions. Example: “The result shows that even under more challenging conditions, our method maintains a stable advantage, suggesting the improvement is not confined to simple scenarios.”

One‑Sentence Takeaway

Do not simply say “the experiment cannot be done”; instead, detail the full experiment’s cost, provide a representative result that answers the reviewer’s core concern, and explain its relevance, noting plans to include full results later.

Project Link

GitHub repository: https://github.com/MLNLP-World/Paper-Rebuttal-Tips