Distinguishing Model Capability from Agent Capability: Frameworks, Benchmarks, and Practical Exercises

Model Capability vs. Agent Capability

Large language models (LLMs) possess static knowledge and reasoning abilities – they answer "what does the model know?" – while AI agents are built on LLMs to perform dynamic tasks in real environments, answering "what can the agent do?".

Model Capability (Model Capability)

Key question: "What does this model know?"

Basic language understanding : semantic comprehension, common‑sense reasoning, knowledge breadth – typical benchmark MMLU , C‑Eval .

Instruction following : ability to correctly interpret and execute user commands – evaluated with task‑specific definitions.

Static task solving : reasoning and output for fixed problem types (e.g., math, code) – benchmarks GSM8K , HumanEval .

Agent Capability (Agent Capability)

Key question: "What can this agent do?"

Autonomy : can the agent decompose goals, plan paths, and execute tasks without step‑by‑step human guidance?

Reasoning ability : can the agent maintain coherent reasoning chains across multi‑step tasks?

Tool usage : can the agent select and invoke external APIs, databases, etc., with correct parameters?

Environment interaction : does the agent perceive feedback and adjust behavior accordingly?

Memory & context : can the agent keep information consistent across multiple dialogue turns and resolve conflicts?

Multi‑agent collaboration : can multiple agents coordinate and share tasks efficiently?

Agent Capability Levels (Shanghai AI Industry Association)

L1 – Basic : passive response, no autonomy, follows preset workflow.

L2 – Assisted : autonomous within predefined scenarios, cannot deviate from the workflow.

L3 – Autonomous : full‑process closed‑loop – task decomposition → path planning → execution → result delivery.

L4 – Collaborative : cross‑scenario coordination with other agents or humans, self‑iteration capability.

Analogy: School Exam vs. Job Performance Review

Model evaluation is like a school exam – a single answer correctness check (e.g., MMLU, GSM8K). Agent evaluation resembles a job performance review – it assesses autonomy, reasoning, tool usage, and handling of unpredictable situations.

Four‑Layer Agent Assessment (Google Cloud)

Output layer : final deliverable – "Did the task finish? How good is the output?" – metrics: task completion rate, quality score.

Reasoning layer (Chain of Thought) : evaluate logical steps – "Are the reasoning steps reasonable and consistent?" – analysis of reasoning chain.

Tool layer : tool invocation – "Were the right tools chosen? Were parameters correct? How efficient was the call?" – tool‑call correctness, API trace.

Memory layer : context consistency – "Can the agent remember prior information? Resolve conflicts?" – multi‑turn interaction consistency tests.

CapaBench Four‑Component Evaluation

Planning module : task decomposition and prioritisation – assess reasonableness of breakdown.

Reasoning module : logical and causal analysis – check chain completeness and decision path.

Action module : execution against environment state – verify correct operation.

Reflection module : error identification and self‑correction – evaluate failure‑recognition and optimisation.

Key Insight

A model that scores highly on MMLU may perform poorly as an agent because knowing information does not guarantee the ability to act. Therefore, model capability and agent capability must be evaluated separately.

Practical Exercise

Participants receive a specification for an intelligent‑customer‑service agent (GPT‑4 base, passed MMLU and GSM8K) with APIs for order queries, returns, and a RAG knowledge base. In groups they classify a list of test items as either model‑only evaluations or full‑agent evaluations, justify their decisions, and present findings.

Core Conclusions

Model capability is the "engine" of an agent, but a good engine does not guarantee a well‑functioning vehicle.

Agent capability assessment must cover tool use, environment interaction, and end‑to‑end task closure – the "full‑car test".

The relationship between the two is hierarchical, not substitutive.