Key Points for Evaluating AI Agents

From Coze to Evaluation

Coze invests heavily in development, tuning, and collaboration, and the hardest part is proper evaluation. To address this, Coze launched the Compass platform, which provides a flexible evaluation framework that distinguishes Coze from other agent platforms.

Why Evaluation Matters

Agent optimization relies on BadCases, and BadCases depend on thorough evaluation; therefore, evaluation is a core focus of agent development.

Compass Evaluation System

Compass establishes a relatively flexible assessment system that includes multiple evaluation dimensions.

Agent Submodule Evaluation Framework

A more reasonable submodule assessment should be organized around four core dimensions: planning, tool use, self‑reflection, and memory. Each dimension is further broken down according to the agent’s strongest capabilities.

Planning

Classic planning

More realistic planning

Tool Use

Simple function calls (FC)

Multi‑turn / multi‑step FC

Complex parameter mapping

Self‑Reflection

Learning from language feedback

Learning from experience

Memory

Assessing performance over time

Dialogue state tracking

Overall Evaluation of Specialized Agents

Successful agents are currently grouped into four categories: web applications, scientific assistance, dialogue/customer‑service, and programming.

1) Web Application Agents

Multi‑turn interaction

Process‑aware benchmarks

More realistic tasks and multimodal interaction

Simplified static simulators

Testing focuses on the first‑turn effect, while the difficulty lies in improving multi‑turn interaction.

2) Scientific Agents

Scientific conception

Experiment design

Experiment execution

Peer review and feedback

Testing focuses on the scientific conception; the main challenge is executing experiments.

3) Dialogue Agents

Key evaluation aspects include:

Transition from static to dynamic dialogue using LLM‑based user simulation

More complex and realistic scenarios involving APIs, databases, and policies

Finer‑grained assessment beyond final results, emphasizing process dimensions

Increasing automation of data updates

Challenges in evaluating the quality of synthesized data

Testing emphasizes complex, realistic scenarios, while difficulty centers on data updates and synthesis quality.

4) Programming Agents

Important evaluation points are:

Code generation (completion, file creation, project generation)

Public code repositories for training and testing at scale

Automated verification via large‑scale unit tests

Testing focuses on unit‑level code quality; the difficulty is ensuring reasonable overall project orchestration.

General Agent Platform Evaluation

Evaluation should involve:

Building comprehensive benchmarks that require multiple tools and capabilities

Integrating several agent‑specific benchmark suites

Testing focuses on tool‑application ability, while the challenge lies in reasonable multi‑agent orchestration.

Conclusion

L3‑level agents have become mainstream; as interaction with agents grows, the ability to evaluate agent capabilities will itself become a new essential skill.

References

1. Coze: Reshaping Productivity with Agents<br/>2. Survey on Evaluation of LLM‑based Agents