Tool Learning with Foundation Models: Frameworks, Datasets, and Open‑Source Toolkits

The article introduces the emerging field of tool learning, where foundation models are trained to understand human instructions and operate external tools, leveraging their strong semantic understanding, world knowledge, and reasoning abilities.

It outlines the background of tool learning, highlighting the question of whether AI can use tools like humans, and presents the definition of foundation‑model tool learning as the ability of a model to follow human commands to operate tools for task completion.

The content categorizes tool learning into two paradigms: tool‑augmented learning, which uses tool outputs to enhance model performance, and tool‑oriented learning, which focuses on the model controlling tools to achieve goals, exemplified by planning and executing multi‑step tasks such as posting on Twitter.

A generic tool‑learning framework is described, analogous to an MDP, consisting of four components: a tool set (physical, GUI, or programmatic tools), a controller (planning based on user instructions), a perceiver (collecting feedback from the environment and user), and the environment (physical or virtual contexts). The interaction loop involves instruction submission, planning, tool execution, feedback collection, summary generation, and iterative updates until the task is completed.

The article details key sub‑tasks such as intent understanding, tool understanding (via zero‑shot and few‑shot prompting), planning and reasoning (introspective vs. extrospective), and training strategies (human behavior cloning and reinforcement‑learning‑based exploration).

It then presents several open‑source contributions from Tsinghua: WebCPM, an interactive Chinese web‑search QA framework with a 5.5k high‑quality LFQA dataset and over 100k real search actions; BMTools, a modular tool‑learning platform supporting custom Python tools, integration with ChatGPT plugins, local models, and planning frameworks like LangChain, BabyAGI, and Auto‑GPT; and ToolBench, a dataset and benchmark for multi‑tool instruction tuning, accompanied by the fine‑tuned model ToolLLaMA.

Additional related works such as WebGPT, WebShop, Toolformer, and tool creation are briefly surveyed, emphasizing self‑supervised tool discovery and the potential for models to generate new tools.

The article concludes with a curated list of tool‑learning papers and a short Q&A session addressing WebCPM’s filtering, multilingual performance, and differences from WebGLM.