How to Respond When Reviewers Claim Your Experimental Setup Is Unfair

In peer review, reviewers often raise the objection “the training settings, hyper‑parameters or compute budget differ from the baseline, making the comparison unfair.” The article explains that such concerns directly affect the credibility of the paper’s conclusions.

Unhelpful Reply

Many authors answer with a vague statement such as “We have ensured a fair comparison and all settings are reasonable.” The article points out that this lacks verifiable details and does not address the specific items reviewers care about.

Is the data split identical?

Are the number of training epochs the same?

Is the backbone model identical?

Are the evaluation metrics the same?

Is the random seed the same?

Was the baseline run with the configuration recommended in its original paper?

Is there any extra compute budget or hyper‑parameter search advantage?

Recommended Reply

A stronger response should explicitly state which settings are identical, which follow the original method’s defaults, and how the overall compute budget is controlled. An example reply is provided:

Thank you for the comment. Fair comparison is indeed crucial. In our experiments we keep the same data split, training epochs, model backbone, evaluation metrics, and random seed for all methods. For method‑specific hyper‑parameters we use the settings recommended in the original papers or tune them uniformly on the validation set. We also limit the total compute budget so that our method does not gain an unfair advantage. We will add a complete configuration table in the revision, indicating which settings are identical, which follow the baseline defaults, and how the compute budget is controlled.

Key Points of a Fair Response

The article summarizes the six concrete aspects that should be addressed:

Data split consistency : “All methods use the same training, validation, and test splits.”

Backbone consistency : “All methods are built on the same backbone to avoid capacity differences.”

Training configuration : “Same number of epochs, batch size, optimizer, and learning‑rate schedule.”

Random seed : “Runs are performed with the same seed and we report mean and variance over multiple runs.”

Hyper‑parameter search fairness : “Key hyper‑parameters are selected on the same validation set with identical search budget.”

Compute budget comparability : “No extra training data or larger compute budget is used; overall training cost is comparable to the baseline.”

One‑Sentence Summary

Fair rebuttal must detail data split, training epochs, model, random seed, hyper‑parameter search, and compute budget rather than merely claiming “the settings are reasonable.”