How SkillCraft Shows AI Agents Can Cut Compute Costs by Up to 80%

SkillCraft is a benchmark introduced by Oxford University together with leading research labs to test whether intelligent agents can autonomously compose basic tools into reusable "Skill" modules, a capability that can reduce computational overhead by as much as 80%.

Why current tool‑calling is inefficient

In real‑world scenarios, tool calls often belong to long‑running workflows filled with repetitive sub‑tasks such as searching data, analyzing content, and extracting summaries. This leads to two major inefficiencies: redundant state passing that wastes massive compute resources, and context‑window saturation where lengthy tool‑call logs quickly fill the model's memory, causing loss of goals or key cues.

Design of the SkillCraft benchmark

The benchmark was built through a rigorous three‑stage pipeline:

Exploration stage – massive testing of existing frontier platforms to identify stable interfaces and task types.

Seed‑task creation – focusing on reliable public APIs and local data to craft benchmark tasks covering weather forecasting, video data scraping, code‑repository analysis, and more.

Systematic scaling – increasing difficulty along two axes: quantity scaling (e.g., analyzing one repository vs. one hundred) and complexity scaling (adding more tool‑call steps per operation).

The final SkillCraft suite contains 126 challenging tasks spanning six application domains and six difficulty levels, providing a solid testbed for assessing the true evolution of large models.

Skill mode evaluation protocol

Based on the MCP framework, the protocol exposes only four lightweight commands: save_skill – persist a successful workflow as a reusable skill. get_skill – retrieve stored skill code and metadata. list_skills – enumerate available skills for a new task. execute_skill – run a stored skill as a high‑level tool.

When faced with a new task, a model first checks the skill library; only if no suitable skill exists or execution fails does it fall back to raw tool calls. Candidate code must pass three validation stages: syntax checking, runtime error reporting (with full stack traces), and post‑execution quality detection (rejecting code that produces >50% unknown or empty outputs).

Empirical findings: flat skill mode

Across a broad evaluation of open‑source and closed‑source models, the Skill mode yields striking efficiency gains. For example, GPT‑5.2’s task success rate rises from 87% to 90% while average compute consumption drops from 1.23 M tokens to 0.26 M (≈79% reduction) and cost falls by 75%. Claude 4.5 Sonnet improves success from 94% to 96% and cuts compute by 71%.

Correlation analysis shows a strong positive link between skill‑execution success and overall task success (0.65) and between compute‑saving magnitude and baseline success (0.53), confirming that more capable models are better at abstracting and reusing high‑quality skills.

Hierarchical skill mode: when deeper nesting hurts

The hierarchical mode unlocks up to ten levels of nested skill calls, theoretically allowing complex compositions. In practice, performance degrades: GPT‑5.2’s success rate falls from 90% to 79% and compute consumption rebounds from 0.26 M to 0.60 M tokens. The failure stems from tiny bugs in low‑level skills (e.g., missing validation for rare dog‑breed personality fields) that propagate upward, causing type errors and cascading crashes in higher‑level skills.

These results suggest that, at the current stage, flat, well‑tested skill libraries are far more reliable than deep hierarchical networks.

Cross‑model skill transfer experiments

Researchers examined whether skills created by one model can benefit others. High‑quality skills from Claude, when executed by Gemini, yielded a 69.2% compute reduction—far exceeding Gemini’s own 14.8% saving with its native skills. Conversely, skills generated by the weaker Minimax model increased compute consumption for other models.

Overall, models that excel at crafting reusable, high‑quality skills add more value than those that merely execute raw tool calls.

Conclusion

The study concludes that, for now, a shallow, rigorously validated skill repository outperforms complex hierarchical compositions. Future multi‑agent systems should prioritize strong models that can distill robust skills, ensuring each computation is spent on the most effective operations.