Can Your AI Skill Pass? An 8‑Dimension Quantitative Evaluation Framework

Skill is the smallest encapsulation of an AI agent’s capability, combining domain knowledge, workflow, and tool integration. While many developers publish Skills, their quality is hard to measure objectively. This article proposes an eight‑dimension quantitative evaluation framework to turn subjective impressions into concrete scores.

Eight Evaluation Dimensions

D1. Metadata Quality : Assesses the description’s precision, keyword coverage, and explicit non‑trigger conditions. It determines whether a Skill can be discovered at all.

D2. Execution Guidance Clarity : Evaluates whether the Skill clearly tells the agent what steps to take, when to ask for clarification, and what actions are prohibited.

D3. Domain Knowledge Density : Measures the amount of specialized knowledge embedded in the Skill (e.g., private APIs, data models, best‑practice guidelines). If the knowledge is easily obtainable elsewhere, the Skill’s existence is questionable.

D4. Workflow Completeness : Checks end‑to‑end flow, step‑to‑step continuity, and handling of exceptions such as API timeouts or file‑download failures.

D5. Input/Output Clarity : Verifies that the Skill clearly states what the user must provide and what the final result will be.

D6. Resource Utilization : Looks at whether scripts, reference documents, and examples are placed appropriately, following a progressive disclosure pattern.

D7. Writing Quality : Assesses the structure, redundancy, and readability of the Skill’s markdown documentation for another AI.

D8. Scope & Focus : Determines whether the Skill concentrates on a single well‑defined task rather than trying to do everything.

Weighting and Rating

Not all dimensions are equally important. We assign weights based on each dimension’s impact on the Skill’s actual effectiveness. After scoring each dimension, a weighted sum is mapped to grades (A, B, C, D, etc.). The detailed scoring formulas are omitted but are available in the appendix.

Real‑World Evaluation Example

Using the framework, the author evaluated a production Skill called ip‑bill (a public‑IP billing workflow). The assessment highlighted specific problems:

D1. Metadata Quality (3/10) : Description was too brief and lacked trigger keywords, causing the agent to miss activation opportunities. Recommendation: expand the description with trigger conditions and a functional overview.

D7. Writing Quality (7/10) : Duplicate content reduced readability. Recommendation: remove repeated SQL examples and merge redundant paragraphs.

The overall strengths were a rigorous workflow, rich domain knowledge, and well‑organized resources. The main improvement area was metadata description and documentation conciseness.

Comparative Evaluation of Two Similar Skills

Two Skills from the dodo catalog— workos‑weekly and subordinate‑weekly‑report —were scored side by side.

workos‑weekly strengths : higher domain‑knowledge density, richer resource links, and a more complex nine‑step workflow.

subordinate‑weekly‑report strengths : superior metadata quality, clearer input/output examples, and broader scenario coverage (six actions).

subordinate‑weekly‑report weaknesses : lower domain‑knowledge density and fewer supporting resources.

Both Skills shared clear trigger conditions, solid execution guidance, error handling, focused scope, and high writing quality.

Multi‑Model Cross‑Validation

Single‑model scoring can be biased; different models may assign different grades to the same Skill (e.g., GLM‑5.1 gave 7.8/A while Claude Opus 4.6 gave 6.5/B). To mitigate this, a three‑stage cross‑validation process is introduced:

Independent Evaluation : Each model scores the Skill on all eight dimensions.

Cross Review : Models view each other’s scores, flag dimensions with a gap ≥ 2 points, and provide evidence from the Skill’s markdown to justify a revised score.

Arbitration : A designated main model aggregates the independent scores and cross‑review comments to produce the final rating, marking dimensions with consensus or dispute.

If the environment does not support multiple models, a fallback uses a single model playing three reviewer roles (strict, pragmatic, and balanced) to simulate diversity.

Four Execution Strategies and Automatic Routing

Different AI tools support different model sets. To ensure the Skill runs across various environments, four execution strategies are defined:

Strategy A: All models are native to the tool.

Strategy B: All models are third‑party; both A and B are used when mixed.

Strategy C: Automatic downgrade to a safe mode when runtime errors occur.

Strategy D: (implicit) combines the above with a Python script that calls the Baidu Qianfan API for third‑party models, handling parallel calls and retries.

The routing logic automatically selects the appropriate strategy based on the provided model list.

Conclusion

The eight‑dimension framework provides a systematic way to assess a Skill’s documentation and design quality, turning “feels okay” into a quantifiable judgment. Multi‑model cross‑validation reduces single‑model bias, though it increases token consumption and evaluation latency. The framework focuses on static documentation quality rather than runtime performance, and future work may include automated static checks, dynamic scoring from execution data, and community‑driven rating systems.