Beyond One-Size-Fits-All: Tailored Benchmarks for Efficient Evaluation

The paper "Beyond One-Size-Fits-All: Tailored Benchmarks for Efficient Evaluation" introduces the TailoredBench framework, which addresses the high cost and distribution‑shift problems of traditional LLM benchmarking by constructing model‑specific evaluation subsets.

TailoredBench follows a four‑step pipeline: (1) use a global probe set (G‑set) to capture prediction features of target models; (2) select a high‑consistency “exclusive” source‑model set for each target; (3) generate a compact N‑set for the target via an extensible K‑Medoids clustering; (4) apply calibration to recover full‑benchmark performance from the reduced set.

Extensive experiments on five NLP and multimodal benchmarks covering over 300 models show that, under the same inference budget of 20–40 queries, TailoredBench reduces MAE by an average of 31.4 % and achieves up to 300× inference efficiency gains, while consistently outperforming baselines such as Random, AnchorPoints, and GP‑IRT in Kendall’s τ.

Ablation studies confirm the importance of Manhattan distance for similarity measurement, the calibration step for accurate score restoration, and the balance of probe set size (optimal around 10 probes). Analyses of source‑model quantity and consistency further demonstrate that both larger exclusive source sets and higher source‑target agreement improve evaluation accuracy.

The framework is adaptable: when new models arrive or inference budgets change, TailoredBench can update estimates without re‑evaluating the entire benchmark, offering a scalable, cost‑effective solution for rapid LLM iteration.