Alibaba's skill-up: Automated Agent Skill Evaluation & Regression Testing

Introduction

skill-up is an open-source Agent Skill evaluation and evolution tool from Alibaba, written in Go. It acts as an evaluation layer on top of Skills: after writing a Skill, developers can codify expected behavior into test cases, then run them on every change to detect regressions or capability degradation instantly.

Core Design Highlights

Declarative Configuration : Environment, engine, model, test cases, and judgment rules are all defined in eval.yaml and cases/*.yaml files.

expect + judge Dual-Layer Judgment : A zero-cost local check ( expect) filters obvious failures first; only passing cases proceed to the more expensive judge step (LLM-based), reducing model flakiness in CI.

Multi-Engine Replay : Built-in support for claude_code, codex, qodercli, and qwen_code — the same test suite runs across different agent engines.

Structured Reports : Outputs Anthropic-compatible grading.json, JUnit XML, or HTML reports; exit codes integrate directly with CI gate checks.

Installation

Install skill-upper (the Skill)

Run in the project directory:

npx skills add https://github.com/alibaba/skill-up/tree/main/skills/skill-upper

The command prompts to select the installed coding agent (e.g., Claude Code). After installation, open Claude Code and type /skill-up; if it completes to /skill-upper, the Skill is ready.

Install skill-up CLI

Download the CLI from the project's Releases page, extract, and verify: skill-up.exe --version The CLI is the actual evaluation runner; the Skill drives it during conversation.

Usage Walkthrough: Evaluating claude-vision-skill

The author tests claude-vision-skill (a Skill that adds image recognition to text-only models) through three iterative rounds.

Round 1: Let skill-upper Create Initial Tests

Prompt used in Claude Code:

使用 skill-upper 评测这个 claude-vision-skill。
阅读 SKILL.md，识别最重要的能力，创建真实的 eval 用例并选择合适的 Judge，
校验配置后运行 skill-up。最后总结结果和影响最大的失败项。

skill-upper did not produce a report immediately. It identified that the current cases could not test the Skill's core capability and recommended adding three scenarios: local image path, --url remote image, and a deliberately non-existent path.

Round 2: Re-test with a Real Image

The author provided a local architecture diagram of an e-commerce mall project (dense information). The Skill described the image, then verified each component. Results showed core capability worked: MySQL, Redis, Docker, ELK, Chinese descriptions, layered structure, and tech stack were all correctly recognized.

Round 3: Two Real Defects Discovered

While core capability passed, skill-upper uncovered two previously unnoticed defects:

max_tokens Hardcoded to 1024 : When asked for a 详细描述 (detailed description), output truncated halfway through the image — missing marketing module, permission module, and data storage. The issue only surfaces with information-dense images.

dotenv Pollutes stdout : Every run prefixes the actual description with two lines of injected env ... logs, contaminating the output and interfering with evaluation itself.

skill-upper provided impact analysis and follow-up suggestions: defect 1 causes truncation on dense images; defect 2 pollutes evaluation logs. It recommended codifying both as permanent regression test cases.

This completed the full loop: 评测→诊断→补用例→重跑 (evaluate → diagnose → add cases → re-run), turning subjective "I think it works" into objective, repeatable verification.

Summary

skill-up transforms Skill quality from gut feeling into explicit, runnable expectations. The standout feature is skill-upper — a Skill that tests other Skills. In this real trial, it found two subtle bugs the author missed, analyzed their impact, and proposed regression tests. Teams developing and maintaining Skills should consider adopting it.

Project Link

https://github.com/alibaba/skill-up