A New Paradigm for Teaching Agents Tools: Insights from ACL 2026 ToolCPT

What problem it addresses

Post‑training fine‑tuning of agents is limited by the scarcity and imbalance of tool data. Table 1 shows that publicly available tool corpora contain at most ten‑thousand distinct tools, and in the Toucan dataset 36.41 % of tools are never invoked while the average tool is called 62 times.

How ToolCPT works

ToolCPT follows three steps.

Step 1 – Mining real tools from code

The authors collected 4.7 × 10¹² tokens from 1.4 × 10¹¹ source files, filtered for self‑contained Python, Go, JavaScript and TypeScript files, and deduplicated to 1.835 × 10¹¹ tokens. Using six agent‑tool criteria (clear API, external information access, environment modification, autonomous execution, explicit feedback, real‑world scenario) they first screened with Qwen‑3‑B and then re‑ranked with GPT‑OSS‑120B, yielding 5.1 million “proxy tools”.

Step 2 – Generating a playbook for each tool

For every proxy tool a playbook is created covering (1) basic information (parameter constraints, return format, mitigation of parameter hallucination), (2) background knowledge (when to use or avoid), (3) call method with at least three correct and three incorrect examples, (4) tool collaboration (compatible or mutually exclusive tools), and (5) error analysis (expected feedback on failures).

Step 3 – Injecting tool knowledge into pre‑training

The tool code, interfaces and playbooks are arranged in ten template orders to form “science‑article‑style” documents with a Chinese‑to‑English token ratio of 1:2, producing 18 billion tokens of tool knowledge. Instead of fine‑tuning, this corpus is mixed at roughly 10 % of the regular pre‑training data during the continuous pre‑training (CPT) phase.

Evidence

Tool authenticity. Human annotation of 1,224 real MCP tools (out of 10,901) shows a cosine similarity of 94.5 % between proxy and real tool category distributions, far higher than the 28.6 % similarity of synthetic tools. t‑SNE visualisation confirms that proxy tools overlap with real tools, while synthetic tools form a separate cluster. Moreover, 95.1 % of synthetic tool names follow a “verb_noun” pattern, 38.7 % start with Get_, indicating models learn naming conventions rather than functional knowledge.

Performance gains. Adding 10 % tool knowledge to CPT and performing lightweight instruction alignment improves eight agent benchmarks. Qwen2.5‑7B’s average score rises by 7.15 percentage points, Llama 3.1‑8B by 5.68 points. Crucially, the same tool data used only for fine‑tuning yields a modest +1.68 point gain, highlighting the benefit of pre‑training integration.

Ablation study. Table 4 shows incremental contributions: using only code achieves 20.62 % of the full gain, adding playbooks raises it to 22.00 %, and including the ten template permutations reaches 23.71 %. General language abilities (MMLU, GSM8k, BBH) remain stable, confirming that tool data are high‑quality code plus knowledge text.

Limitations. The optimal tool‑to‑regular‑data ratio is around 10 %; increasing to 20 % reduces the gain to 22.96 % because excessive task‑specific data dilutes other signals. Proxy tools are static simulations, and the synthetic error feedback cannot capture dynamic environment states. Multi‑tool long‑range collaboration remains unsolved. The mining process required 350 k GPU‑hours, making replication difficult for most teams.

ToolCPT: Improving Tool Utilization in LLM Agents via Continuous Pre-training
https://aclanthology.org/2026.findings-acl.776.pdf