Are We Over-Testing LLMs? Survey of Benchmark Compression Methods, Costs & Reliability

Introduction

Large language models iterate rapidly, but evaluation grows heavier: each version runs full test suites and invokes expensive judges or human review. Researchers from Sun Yat-sen University survey 119 papers in Are We Testing Too Much? A Survey of Benchmark Compression for Efficient Evaluation of Large Language Models (arXiv:2610.18248) to systematically discuss how partial evaluation can approximate full evaluation scores, rankings, or model-selection results at lower cost.

The survey organizes research into static benchmark compression, adaptive evaluation, and benchmark analysis with foundations, comparing information requirements, result recovery, full lifecycle costs, and failure conditions. It does not advocate uniformly deleting a fixed proportion of items, nor does it equate "testing less" with "evaluating accurately." Reliable compression must first define what results to preserve, then verify coverage, estimation, and decision reliability.

Problem Formulation and Taxonomy

2.1 Problem Definition

Full evaluation fixes the model, protocol, evaluation units (usually items), and aggregation rule. Compressed evaluation executes only a subset of units, possibly using item features, historical responses, or cheap signals from the target model, then applies a recovery rule to estimate full results. Two conditions must hold: result difference stays within a specified tolerance, and the compressed evaluation is cheaper under the same cost accounting.

Thus evaluation units, recovery method, and error target must be defined together. Preserving overall ranking does not equal precisely recovering each model's score; identifying the winner does not equal recovering the full ranking; a short benchmark's average does not naturally represent the original benchmark's average. When generation, judging, or agent runs involve randomness, one must distinguish "recovering a specific full run" from "estimating expected performance."

2.2 Taxonomy Overview

The taxonomy follows a sequential decision process, not a flat attribute list. First, does the work primarily reduce evaluation cost or study redundancy, measurement, and reliability? The latter goes to benchmark analysis and foundations. Second, after main evaluation starts, can existing results change subsequent testing, budget, or stopping point? If yes, it is adaptive evaluation, split into item selection or sequential decision. For static methods, does it jointly reduce effort across multiple benchmarks? If yes, cross-benchmark compression; otherwise single-benchmark compression. Single-benchmark methods are further divided by required information: content-driven, historical-response-driven, and target-model-conditioned. This yields six operational method families plus a foundational research category. Each paper receives a primary label; execution timing, scope, information source, saved units, and output type are auxiliary attributes. The survey notes that applying the same method to multiple benchmarks independently is not cross-benchmark joint compression, and using a probe to pick a fixed batch is not continuous adaptation during main evaluation.

Static Benchmark Compression

3.1 Single-Benchmark Compression

Single-benchmark methods fix a subset before main evaluation and use its results to estimate the full benchmark's performance. Static does not require all models to use the same subset: if a cheap signal from the current model participates in selection, each model can have its own fixed subset.

3.1.1 Content-Based Compression

These methods use random or stratified rules, and item text, metadata, and representations to build subsets, without requiring a historical model-item response matrix. Random sampling can estimate the overall mean; stratified sampling aggregates with group weights; PromptEval predicts unobserved units in a sparse design of prompts and exemplars. Core-set methods group by text, required skills, or reasoning structure, select representatives, then use weights or prediction to recover results. Some methods leverage content-assisted scoring to draw more informative items and correct uneven selection with inverse-probability weights. Advantages: low startup cost, no need to fully evaluate many historical models first. Risks: small random subsets have high variance; diversity in embedding space does not guarantee coverage on capability dimensions. Weak reliance on historical patterns may improve robustness under distribution shift, but cannot guarantee accuracy for any model.

3.1.2 History-Response-Based Compression

These methods reuse existing model responses across items to form a model-item matrix. Similar response patterns reveal redundancy; discriminative items help estimate new model capabilities. Anchor Points, tinyBenchmarks, etc., use clustering or item response theory to map limited responses to full scores; MiniLongBench, SparseEval, etc., use representative points and prediction mechanisms to recover untested results. Others combine random sampling with residuals or inverse-probability correction to keep the original benchmark as the estimation target. Historical records can yield smaller subsets or more accurate estimates, but building the matrix often requires multiple full evaluations. After new model families or large capability jumps, old response relationships may break, and random sampling may remain competitive. Historical coverage, construction cost, and transfer validation cannot be omitted. A related class of historical filtering retains hard or high-discrimination items but directly scores the new benchmark without recovering the original full score. This is a boundary case: if the target changes, it should be treated as benchmark revision or short-version validation, not packaged as faithful compression.

3.1.3 Target-Model-Conditioned Compression

Target-conditioned methods first obtain a low-cost signal from the current model, then fix the main evaluation batch. TailoredBench uses shared probes to find similar historical models and builds a dedicated subset for the target; uncertainty or disagreement signals can also guide item selection and inter-model comparison. Another path uses a cheap judge to evaluate a larger pool, then sends a fixed small portion to human or strong judges to audit agent errors and correct the full estimate. Here the savings may be in human labels or strong-judge calls, not in target model generation. Gains depend on whether probes, full-pool scans, or cheap scoring are far cheaper than the saved test cost. Weak signals may miss critical items; proxy bias may systematically affect results; random coverage and paired human verification can provide additional safeguards.

3.2 Cross-Benchmark Compression

Multiple benchmarks may measure overlapping capabilities. Cross-benchmark compression jointly selects items and shares information to recover per-benchmark scores, suite averages, or suite rankings, rather than independently compressing each benchmark. Suite-anchor methods pick high-information items across tasks and use a shared capability model to recover results; benchmark-selection methods run a few full tasks and use the score matrix to predict skipped tasks; source-to-target methods use cheaper tasks as proxies for expensive benchmarks, estimating target performance via regression or importance weighting. Multilingual tensor models further share structure across languages. The savings unit expands from single items to entire benchmarks or expensive environment runs, but risks also scale: relationships between benchmarks may not hold for new models. Validation on unseen models is required, and portions that cannot be reliably estimated should be run directly. If the proxy source task's cost approaches the replaced evaluation, one cannot claim savings merely by "skipping the target task."

Adaptive Evaluation

4.1 Adaptive Item Selection

Adaptive item selection starts with a small batch, updates score, ability, or uncertainty after each response or batch, then chooses the next most informative test. Unlike static methods, main evaluation results alter the subsequent path. Computerized adaptive testing uses item response models to select the next item that provides more ability information; active matrix completion selects items that help predict unobserved responses. Other methods concentrate budget on uncertain groups, rare failures, or noisy judge units, correcting with group weights, sampling probabilities, or repeated tests. These methods use samples more effectively but incur two costs: serial rounds may increase actual wait time, and different models' test paths weaken direct comparability and reproducibility. Samples biased toward hard or uncertain items no longer naturally represent the full benchmark, so matching estimation correction, historical calibration, and full path logs are needed. Better item selection does not equal knowing when to stop. The information value of the next test and the reliability of the final decision are different problems.

4.2 Sequential Decision and Stopping

Sequential methods periodically judge whether to continue testing, targeting score precision, reliable comparison of two models, or finding the best model under a fixed budget. They may also accept, reject, escalate to a strong judge, or defer. Flash-HELM gradually expands samples and stops when leaderboard stratification is sufficiently clear; CELEUS combines direct testing to correct predictions and decides continuation based on interval precision; UCB-E concentrates a fixed budget on potential winners. The latter reallocates budget but does not imply reducing the predetermined total budget. Sequential rules must be valid under repeated looks at results. Standard fixed-sample intervals cannot serve as anytime-stopping guarantees; methods supporting valid checks at arbitrary times rely on corresponding statistical conditions. Advantages: spend tests where they can still change decisions. Limitations: close models may need near-full evaluation; phased execution may reduce parallelism; some methods only return a winner or threshold judgment, not all scores. When stopping conditions are not met, one should continue, report indecision, escalate review, or abstain.

Benchmark Analysis and Foundations

Why can we compress? Items may measure overlapping capabilities, model responses are highly correlated, some items have low discrimination, and the overall response matrix has limited effective dimensionality. Few representative items can therefore retain main information. However, redundancy depends on the current model population and output target; preserving overall ranking does not guarantee preserving score gaps, especially for very close models.

How many items can be deleted? No universal safe proportion exists. Required sample size relates to the score difference of interest, pairwise variance, capability coverage, and allowed error rate. Item response theory further needs a well-calibrated model and stable item parameters; parameter uncertainty affects selection and ranking. When evidence is insufficient, comparisons should remain undecided rather than forced into precise rankings.

Which conditions must be fixed? Full and compressed evaluations should use the same judge, prompts, decoding settings, and agent run frameworks. Overall judge accuracy may mask local errors; prompt or framework changes can rewrite model comparisons. The impact of protocol changes must be evaluated separately from compression itself.

Compression or repair? Deleting items with answer bias, measurement errors, or anomalous capability ordering may yield a cleaner benchmark, but scores and rankings may also change. Compression seeks to approximate original results; repair asks whether original results are worth preserving. A repaired smaller benchmark should be independently validated as a new benchmark.

Comparison Across Method Families

6.1 Subset Selection: Information and Timing

Content and history methods can select fixed subsets before testing the target model; target-conditioned methods first gather cheap signals, then fix a batch; cross-benchmark methods jointly select in advance. Adaptive item selection uses each new result to decide the next item; sequential decision uses interim evidence to decide whether to continue testing. Later, target-proximal information may improve efficiency but reduces cross-model reuse and parallelism. Selection should first assess what information is on hand and when it can be obtained, not simply pick the most advanced-sounding name. Static main batches do not guarantee the entire two-stage design is statistically non-adaptive: if probes influence selection, probe reuse and estimation handling must be explained, and costs accounted once, not ignored or double-counted.

6.2 Recovering Full Results

Proper random designs make sample means or stratified means correspond to the original population; deterministic core-sets represent items and their cluster weights but do not automatically have the same protection. Inverse-probability weighting requires true valid selection probabilities and sufficient support; extreme weights inflate variance. Historical prediction can recover missing results but depends on transfer to new models. Prediction with residuals or human-label correction uses observed errors to adjust proxy estimates; under the method's prescribed sampling and estimation conditions, poor prediction may reduce precision and savings but does not necessarily invalidate the corrected estimate. Checks must match actual outputs. Scores need error and bias reporting; rankings need correlation and near-tie reversal checks; winner or threshold decisions need corresponding error rates; intervals need both coverage and width; methods allowing abstention must report risk on decided cases and decision proportion. Abstention is not a correct judgment; average score error cannot substitute for tail or quantile reliability.

6.3 Full Lifecycle Cost

True cost includes construction, calibration, target-side signals, expensive tests, recovery estimation, maintenance refresh, audit, and necessary fallback. The total account over multiple evaluations is: initial construction cost + asset refresh cost + sum of per-target run costs. Content methods avoid building full historical matrices but still incur full-pool embedding or inference trajectory costs. History and cross-benchmark methods invest upfront in reusable assets; only when actual repeated use savings offset construction and maintenance does overall benefit appear. Target-conditioned methods pay probe or proxy costs per target; adaptive methods may reduce calls but increase serial wait. One should separately report item counts, model calls, judge calls, human audits, compute, monetary cost, and wall-clock time. Without declared conversion, human time, GPU time, and fees should not be directly added. Proxy evaluation of all outputs then reducing human audit saves labeling work, not target generation compute.

6.4 Limitations and Method Selection

When only item data is available, consider content or stratified sampling; when historical records are broad and protocol-consistent, consider history-response methods; when cheap target signals are truly effective, consider target-conditioned methods. For jointly evaluating related suites, use cross-benchmark methods; when serial rounds are allowed and estimation can handle the selection process, use adaptive item selection; when a final decision is explicitly needed and valid stopping rules exist, use sequential methods. Validation must cover the deployment transfer scope. Leave-out-model tests known tasks on new models; leave-out-family checks family transfer; time leave-out tests future versions; task leave-out validates tasks not used in construction. Leave-out must cover construction, fitting, hyperparameter tuning, and calibration; one cannot first use all target results to select items, then hide only part of the matrix and claim new-model generalization. Offline replay and online testing can use the same leave-out splits, but adaptive replay can only reveal results already obtained on the current path, not peek at the future. Prospective tests should freeze the process before new targets arrive; theoretical guarantees only apply to their declared targets, randomness, and assumptions. Fair comparison should align reference protocol, output, budget, and resources, including random or stratified baselines, comparing error under same budget or cost under same quality, and reporting variation across multiple random runs. If conditions are not met, one should expand random coverage, directly test unsupported tasks, continue evaluation, or abstain, rather than insist on maintaining a compression ratio.

Challenges and Future Directions

Cross-model and cross-time generalization. Compressed subsets often come from existing model responses; new families or stronger models may change difficulty and discrimination patterns. Future work needs more robust transfer and continuous checking of coverage and discrimination on newly released models, refreshing when they drop. Validation only on random splits of old models hardly answers applicability to future versions.

Updating existing benchmarks. Official leaderboards and historical papers still reference full results, so compression will likely first serve development iteration, screening, and repeated verification. Combined with benchmark repair, releasing recalibrated versions that remove flawed, redundant, too-easy, or low-discrimination items can create cheaper yet informative public references. However, new versions cannot be mixed unconditionally with old scores; they need re-validation of measurement goals and comparability.

Considering compression before release. For new benchmarks, construction should balance compactness, informativeness, and capability coverage, rather than releasing large redundant sets and letting each user compress independently. This reduces evaluation and model iteration costs and avoids unnecessary compute. Compact design still needs model populations, error targets, and subsequent maintenance evidence; it cannot pursue minimal item count alone.

Conclusion

The survey's unified perspective: use representative tests, cheap or reusable information, and additional evaluation for uncertain cases to retain reliable results while reducing cost. It simultaneously reminds us that reliable compression requires sufficient coverage, accurate recovery, and effective adaptive decisions. What a method saves, what upfront investment it needs, whether it holds for new models, and how to fall back on failure must be answered together. For practitioners, the most valuable sequence is not "delete items first, then check error," but first fix evaluation goals and protocol, clarify available information and total budget, then verify required outputs, and resume testing where evidence is insufficient. More efficient evaluation gathers enough useful information at lower cost, not faster unsupported conclusions.

Limitations of the Survey

The authors note three limitations: literature scope ends August 2026, so later work is not covered; space limits the main text to taxonomy and roadmap, with details in appendices; different studies vary widely in models, tasks, costs, and protocols, making direct quantitative merging difficult. Therefore the survey does not compile a unified leaderboard of compression ratios under different conditions. Appendix reading guide: Appendix A lists paper catalog and category boundary cases; B compares adjacent method surveys and general evaluation surveys; C gives scope, taxonomy rules, decision paths, and auxiliary attributes; D expands result definitions, selection and recovery, information, cost, validation, and selection; E explains representative methods, assumptions, failure conditions, and fallbacks per category; F describes retrieval, screening, and version merging; G discloses human-review tooling and impact; H summarizes survey contributions. Readers needing to reproduce selection and verify taxonomy should focus on C, D, E, not just retained sample ratios.

Original paper: https://arxiv.org/abs/2610.18248; PDF: https://arxiv.org/pdf/2610.18248.

Code example

来源：专知
本文
约7000字
，建议阅读
10
分钟
不主张统一删掉某个比例的题目，也不认为“测得少”就等于“评得准”。