PACE: Predicting Agent Performance with 1% of Evaluation Cost
The PACE framework predicts agent performance on expensive benchmarks like SWE-Bench and GAIA by selecting predictive questions from cheap non-agentic benchmarks, achieving under 4% prediction error, 0.80+ Spearman correlation, and 85% pairwise ranking accuracy at less than 1% of full evaluation cost, enabling a two-layer evaluation strategy for efficient model selection and iteration.
The High Cost of Agent Evaluation
Evaluating agent capabilities is becoming increasingly expensive. While testing a base model's QA, coding, or reasoning is relatively cheap, agentic benchmarks like SWE-Bench and GAIA require complex environments, toolchains, long rollouts, multi-turn calls, code execution, retrieval, verification, and error recovery. A single full evaluation can cost hundreds to thousands of dollars.
In real development, evaluation is not a one-time event: models change, prompts are tweaked, toolchains adjust, memory strategies evolve. Running full agent benchmarks for every iteration quickly becomes prohibitive in cost and time.
Complex Agent Tasks Rely on Atomic Capabilities
Although agent tasks appear complex (e.g., fixing a bug in a real repository involves understanding issues, browsing code, locating problems, modifying files, running tests, and iterating on failures; answering complex questions requires retrieval, evidence selection, tool use, multimodal integration, and verification), they decompose into a set of fundamental capabilities:
Instruction following
Task planning
Code generation
Tool calling
Information retrieval
Multimodal understanding
Error recovery
Verification and testing
PACE's core idea is to treat agent capability not as a black box but to predict overall performance on complex agent tasks using cheaper, easier-to-measure slices of these atomic capabilities.
Selecting the Most Predictive Questions from Cheap Benchmarks
PACE does not simply average a collection of small tests. Instead, it selects a small subset of questions from a large pool of low-cost non-agentic benchmarks that are most predictive of performance on agentic benchmarks. The model's results on these selected questions are then used to predict its scores on high-cost benchmarks like SWE-Bench and GAIA. This curated set forms PACE-Bench.
Experiments covered 14 models, 4 agentic benchmarks, and 19 non-agentic benchmarks. Key results:
Prediction error below 4%
Spearman correlation exceeding 0.80
Pairwise model ranking accuracy around 85%
Cost less than 1% of full agent evaluation
This demonstrates that in many scenarios, expensive agent benchmarks need not be run in full for every iteration to make meaningful capability judgments.
Toward a Two-Layer Evaluation Strategy
The value extends beyond cost savings. PACE proposes a more engineering-oriented evaluation approach: shifting from running full tasks every time to estimating overall performance via cheaper capability slices.
For daily development, this enables:
Model selection: Quickly filter out unsuitable models with low-cost proxy evaluation before sending a few candidates to high-cost real tasks.
Model routing: Estimate capability distributions across planning, coding, retrieval, and multimodal strengths to decide which model handles which task.
Agent harness iteration: Use low-cost proxy evaluation as a feedback signal for prompt, tool, memory, retry, and verification strategy changes, avoiding full evaluation on every tweak.
Future: A Two-Layer Evaluation System
Agent evaluation will likely evolve into two layers:
First layer: A few high-cost real tasks for final validation, closely mirroring production environments to test end-to-end complex workflows.
Second layer: Many low-cost proxy evaluations for daily iteration, model screening, and directional decisions. This layer does not replace real tasks but significantly reduces trial-and-error cost.
This mirrors engineering practice: not every code change triggers a full regression suite, and not every model update undergoes the most expensive end-to-end verification. An efficient evaluation system lets low-cost signals cover daily work while reserving high-cost evaluation for critical checkpoints.
Engineering Agent Evaluation as Capabilities Grow
As agents become more capable, task chains lengthen and evaluation costs rise. Relying solely on a few expensive benchmarks for all decisions will increasingly hinder rapid iteration. PACE illustrates that agent evaluation itself must be engineered.
Evaluation should become a continuous feedback system within the R&D process, not just a leaderboard score. Questions such as which capabilities most influence final performance, which small tests best predict large tasks, and which metrics suit daily iteration versus final acceptance will grow in importance.
For teams building agents, a practical recommendation: do not focus only on the final large benchmark; establish your own low-cost capability-slice evaluations. They may not be perfect, but they make model selection, prompt tuning, tool design, and system iteration faster and more controllable.
The competition in agents lies not only in the models themselves but also in evaluation systems. Whoever can judge capability changes more cheaply, reliably, and promptly will iterate faster.
Paper: arxiv.org/abs/2607.02032
Signed-in readers can open the original source through BestHub's protected redirect.
This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactand we will review it promptly.
How this landed with the community
Was this worth your time?
0 Comments
Thoughtful readers leave field notes, pushback, and hard-won operational detail here.
