Skill-up: Open-Source Evaluation Tool for AI Agent Skills

Background

In 2026, Skills have become standard in AI-assisted programming. Developers routinely distill personal experience and team best practices into Skills, but a key question remains: who defines the quality of a Skill, and what are the evaluation criteria? The author searched for a quantifiable evaluation system and discovered Alibaba's open-source tool Skill-up, designed for Agent Skill developers to verify functional correctness in real Agent Engine environments such as Claude Code, Codex, and Qoder CLI, convert failed cases into precise fix guidance, and support continuous regression in local and CI environments.

What is Skill-up?

Skill-up is a command-line evaluation tool for Agent Skills. It expects a specific directory structure:

my-skill/
  SKILL.md          # Your Skill definition
  evals/            # Evaluation directory
    eval.yaml       # Evaluation entry config
    cases/          # Test cases directory
      basic-test.yaml   # A test case
      edge-case.yaml    # Another test case
    fixtures/       # Test resources (optional)
      repos/        # Repository templates
      scripts/      # Evaluation scripts

Installation

Two installation methods are provided:

Via Skill-upper (recommended for Codex)

# Codex, global install
npx skills add https://github.com/alibaba/skill-up/tree/main/skills/skill-upper -g -a codex -y

Via command line (pure CLI)

curl -fsSL https://raw.githubusercontent.com/alibaba/skill-up/main/install.sh | bash

Key Commands

Skill-up offers a rich set of commands for running evaluations:

# Run all cases
skill-up run ./evals/eval.yaml

# Run only matching cases
skill-up run ./evals/eval.yaml --include-case-name "basic-*"

# Exclude specific cases
skill-up run ./evals/eval.yaml --exclude-case-name "*-old" --exclude-case-name "*-deprecated"

# Specify Engine and model
skill-up run ./evals/eval.yaml --engine codex --model openai/gpt-4

# Override case parallelism temporarily
skill-up run ./evals/eval.yaml --parallelism 4

# Enable baseline comparison
skill-up run ./evals/eval.yaml --baseline

# Generate multiple report formats
skill-up run ./evals/eval.yaml --format json --format html --format junit

# Run 3 consecutive iterations, output to iteration-1/ through iteration-3/
skill-up run ./evals/eval.yaml --iteration 3

# Auto-detect mode (consume Anthropic evals.json directly)
skill-up run --auto
skill-up run --auto --engine codex
skill-up run ./my-skill/ --auto

Verification and Demo

After installation, a success screenshot confirms the tool is ready:

Skill-up can also be invoked via Codex:

Creating and Running the First Evaluation

The article demonstrates creating a demo evaluation and shows the test execution:

The resulting evaluation report is displayed: