Best Practices for Skill Evaluation and Optimization with Alibaba Cloud AgentLoop

Introduction

In AI Agent applications, a Skill is the key asset that gives an agent domain‑specific capabilities. To ensure a Skill works well in real scenarios, developers must answer three questions: how to verify its quality, how to quantify performance, and how to continuously improve it.

Overall Process Overview

The end‑to‑end workflow consists of six stages: create the Skill, integrate observability, build an offline evaluation dataset, run offline experiments with multi‑dimensional scoring, analyze Bad Cases, and iterate verification before release. The loop repeats until the overall score continuously meets the target threshold.

Step 1 – Create Skill and Integrate Observability

A Skill is defined by four elements: Prompt instructions, tool‑call specifications, output constraints, and security boundaries. After creation, the Skill remains in draft state and can be tested locally. The article uses the alibabacloud‑cms‑manage Skill as an example.

When a Skill is deployed, AgentLoop automatically collects execution data as Trace Spans, so developers obtain a full execution view without adding custom instrumentation.

Step 2 – Build the Evaluation Dataset

The dataset is a collection of test cases. Each case includes a name, the natural‑language prompt, expected behaviors (CLI commands or API calls), expected outputs, and forbidden outputs (e.g., AccessKey, SecretKey, internal IP). A typical case is shown below:

# A typical evaluation case
case:
  name: "Query Prometheus monitoring data"
  prompt: "Help me view the Prometheus monitoring data in the current workspace"
  expected_behaviors:
    - "arms-prometheus query"
    - "kubectl get pods"
  expected_outputs:
    - "Monitoring data query succeeded"
    - "Workspace information"
  forbidden_outputs:
    - "AccessKey"
    - "SecretKey"
    - "Internal IP address"

Design principles for the dataset are:

Cover real business scenarios : use actual CMS operations such as alarm queries, Prometheus queries, and APM/RUM list‑get chains.

Include both positive and negative examples to ensure comprehensive evaluation.

Make expectations verifiable so the evaluator can automatically judge pass/fail.

In practice, an initial dataset of 10‑20 core scenarios is recommended, each containing at least one positive and one boundary case.

Step 3 – Run Offline Experiments and Multi‑Dimensional Evaluation

AgentLoop provides a full offline experiment engine. For each experiment the system performs:

Environment preparation : launch the selected Agent framework (e.g., Codex, Claude Code) and model.

Case execution : run each test case and record input, output, and any errors.

Contrast experiment : run a control group without the Skill to quantify the Skill’s incremental value.

Multi‑model coverage : execute the same cases on several Agent‑framework/model combinations to assess Skill portability.

The evaluation results are automatically scored on dimensions such as task completion, evidence support, tool‑call success rate, security, hallucination detection, and correctness. Two types of evaluators are available:

Built‑in evaluators : ready‑to‑use metrics covering common dimensions.

Custom evaluators : user‑defined logic for domain‑specific checks.

After the run, AgentLoop generates a comprehensive report containing overall score, per‑case scores, side‑by‑side comparison (Skill vs. no‑Skill), and detailed evaluation traces.

Step 4 – Bad Case Analysis and AI‑Assisted Optimization

Bad Cases are identified by sorting experiment results by low scores. The platform allows filtering by dimension (e.g., security < 0.5) or by a score threshold. The following Python script demonstrates how to query evaluation results via the AgentLoop SDK and extract cases with a score below 0.6:

#!/usr/bin/env python3
"""Fetch evaluation results from AgentLoop and filter bad cases (score < 0.6)."""
import json, os, sys
from agentloop_sdk._vendor.alibabacloud_agentloop20260520.client import Client
from agentloop_sdk._vendor.alibabacloud_agentloop20260520 import models as main_models
from alibabacloud_tea_openapi import models as open_api_models

AGENT_SPACE = "YOUR_AGENT_SPACE"
REGION_ID = "cn-hongkong"
AK = "ALIYUN-AK"
SK = "ALIYUN-SK"
EXPERIMENT_ID = "EXPERIMENT_ID"
SCORE_THRESHOLD = 0.6

def create_client():
    config = open_api_models.Config(
        access_key_id=AK,
        access_key_secret=SK,
        region_id=REGION_ID,
        endpoint="agentloop.cn-hongkong.aliyuncs.com",
    )
    return Client(config)

# ... (functions to list tasks, runs, and fetch results) ...

After extracting Bad Cases, developers can feed the structured data (input, output, score, explanation) to an AI assistant (e.g., Qoder) to obtain root‑cause analysis and concrete Skill modification suggestions. Human reviewers then validate the suggestions and create a new Skill version.

Step 5 – Iterative Verification

Each time a Skill is modified, the entire offline experiment is rerun and the new results are compared with the previous version. AgentLoop tracks version history, score trends, and regression detection. An example version table illustrates continuous improvement:

Version   Correctness   Security
-------------------------------
v0.1        0.99        0.57   (initial draft)
v0.2        1.00        0.49   (add security constraints)
v0.3        0.94        0.68   (add Harness engineering)
v0.4        0.98        0.78   (final release candidate)

Contrast experiments (Skill vs. no‑Skill) remain in place to verify that observed gains are truly due to the Skill and not model variance.

Step 6 – Release and Continuous Monitoring

When the overall score reaches the predefined threshold (e.g., all dimensions ≥ 0.8), the Skill is promoted from draft to a released version and becomes available to all users. After release, the Skill dashboard continues to collect online metrics such as call volume, success rate, and user coverage.

Developers monitor these metrics, capture new real‑world traces, and feed any newly discovered scenarios back into the evaluation dataset, thus starting a new iteration of the loop.

Key Principles

Data‑driven : every optimization decision is backed by experiment data.

Contrast verification : maintain control groups to quantify the Skill’s incremental value.

Closed‑loop iteration : observe → evaluate → optimize → verify, forming a continuous improvement cycle.

By following this best‑practice framework, developers can systematically raise Skill quality, achieve measurable performance gains, and reliably deliver high‑quality AI Agent capabilities.