Building Trustworthy AI Evaluation: Consistency, Confidence & Pluggable Platform

Introduction

AI output quality remains hard to quantify: teams write custom evaluation scripts, standards differ, datasets scatter across repositories, and execution records are untraceable. No one can tell whether a score drop means the AI degraded or the evaluation rules shifted. This article presents an AI evaluation platform built to make scores trustworthy by attaching consistency (scenario stability) and confidence (evaluator stability) to every score, and by using a pluggable architecture that fits diverse evaluation scenarios.

1. Background: Why an Evaluation Platform?

1.1 Problems and Challenges

Every AI application faces the same question before launch: is this agent/skill/knowledge base good enough? Traditional software has clear pass/fail criteria (HTTP 200, test cases pass). AI outputs are open-ended and non-deterministic — same input can yield different results — so there is no ready-made ruler. Most teams stay stuck at "write a script, run a few cases, eyeball a score." The cost of blind launches is real: a silently degraded knowledge base feeds stale or wrong info to all downstream users; an unevaluated code-generation agent pushes errors into production.

Four scaling walls appear when evaluation needs grow across knowledge-base quality, code generation, data construction, recommendation quality, knowledge retrieval, and issue discovery:

Evaluator reinvention: Each new scenario rewrites the full evaluation logic; data formats, calibration methods, and scoring standards differ, preventing cross-scenario comparison and continuous observation.

Dataset chaos: Evaluation data lives in scattered Excel/JSON/YAML files; versions conflict; nobody knows which branch holds the latest test set.

Stability ignored: One run scores 8.5, the next 6.2 — did the skill degrade or did the evaluator wobble? No one knows.

Results don't drive decisions: Reports lack structured data; metrics cannot be quantified to gate releases.

Beyond scale, three fundamental challenges differ from traditional software testing:

Challenge 1: Non-determinism. Same prompt, same data, different output every run. In knowledge-base evaluation, an LLM asked to extract method signatures from docs and compare with source code returns 3 items one run, 8 the next. This is LLM nature, not a bug. Without multi-run execution + statistical analysis, a single score is a dice roll.

Challenge 2: Black-box behavior drift. Model upgrades, prompt tweaks, upstream API changes can silently skew behavior. The nastiest drift is "local degradation, overall score unchanged": a systematic misjudgment on a low-frequency category gets masked in the aggregate. Without scheduled inspections and layered evaluation, such regressions go unnoticed.

Challenge 3: Evaluator uncertainty — "Who evaluates the evaluator?" Using LLMs to judge LLMs inherits biases and drift. Zheng et al. (MT-Bench/Chatbot Arena) showed GPT-4 as judge agrees with human preference >80% (on par with human-human agreement), but also identified systematic position bias, verbosity bias, self-enhancement bias, and math/reasoning scoring failures. Evaluators cannot be trusted unconditionally.

Human review doesn't scale. A knowledge-base domain may have 200+ nodes, each with 5–15 method signatures — thousands of "doc vs. source" comparison points. Humans can spot-check dozens; full comparison is beyond cognitive capacity. A comparison table shows manual review takes days, covers only samples, relies on eyeball matching, cannot guarantee cross-domain consistency, and depends on reviewer experience; the platform completes a full domain in 20 minutes, covers 200+ nodes, uses LLM + source-code diff, enforces unified scoring standards, and is 100% repeatable.

1.2 Industry Research: How the Industry Solves These

Before building, the team surveyed public methodologies, academic research on LLM judges, and mainstream open-source frameworks. Conclusion: industry consensus exists on "how to run one evaluation correctly," but existing tools cannot support "continuous evaluation of dozens of AI application types within an organization" — the gap this platform fills.

Methodology consensus: Anthropic's "Demystifying evals for AI agents" recommends combining code-based, model-based, and human graders; prioritize deterministic checks; reserve human review for limited verification; calibrate LLM judges with human experts, clear rubrics, separated scoring dimensions, and allow "Unknown" when evidence is insufficient. Datasets should start from 20–50 simple tasks derived from real failures; turn bugs and support tickets into test cases. For non-determinism, use pass@k (at least one success in k runs) and pass^k (all k runs succeed); consistency-sensitive products should monitor the latter. Split eval sets into capability sets (low initial pass rate, guides improvement) and regression sets (high pass rate, prevents regression). OpenAI's "Testing Agent Skills Systematically with Evals" echoes systematic skill testing via evals.

Methodology converges: evaluation is an engineering combo of deterministic checks + model scoring + human calibration — far more than "ask a model for a score"; data must come from real bad cases; judges need calibration. But methodology answers "how to do one evaluation right," not "how an organization continuously evaluates dozens of AI application shapes" — that needs a platform.

LLM-as-a-Judge: Validity and Risk Coexist. Zheng et al. gave two answers: GPT-4 as judge reaches >80% agreement with human preference (comparable to human-human), so judges are usable. But the same work systematically identified position bias (judges prefer first answer in pairwise comparison), verbosity bias (prefer longer answers; "repeated-list attack" inflates scores), self-enhancement bias (tend to score own/family-model outputs higher — limited evidence), and math/reasoning scoring failures (judges can be misled even on problems they can solve). Judges cannot be trusted blindly.

This directly motivated the platform's confidence design: since top research proves judges have bias and jitter, the platform must quantify and constrain judge behavior rather than avoid judges. Details in Chapter 3.

Open-source frameworks: What they solve, what they don't. OpenAI Evals (config + dataset driven, custom scorers, multi-turn & tool-use), promptfoo (CLI/YAML prompt comparison & assertions, red-teaming), Ragas (RAG-specific metrics: faithfulness, context recall), DeepEval (unit-test-style LLM assertions). Commercial SaaS platforms bundle evaluation with debugging and observability. Commonality: developer-centric, lightweight for "run one eval in my project." Gaps vs. the four walls: dataset management is file-based, no versioning/grouping; evaluators are fixed assertions/rule templates, cannot handle complex logic like repo cloning and multi-step reasoning; judge confidence is not quantified; scheduled patrols, change triggers, cross-experiment comparison — "continuous operations" capabilities — are mostly missing. Internal businesses also require data compliance and private deployment, ruling out external SaaS.

Differentiation. A side-by-side comparison table highlights the platform's choices: covers four pluggable test-object types (code-gen, agent, knowledge-base, skill) vs. industry's single-point prompt/model/RAG eval; quantifies judge jitter, removes outliers, uses human evaluators as calibration baseline vs. industry's methodology-level bias identification & calibration advice; separates consistency (test-side) and confidence (judge-side) via multi-run execution vs. industry's pass@k/pass^k sampling; unified dataset service + dataset groups + golden sets + recommendation-acceptance loop vs. lightweight dataset loading; manual/scheduled/change-triggered continuous patrol vs. manual one-off runs; structured issue list + severity grading + accept/reject closure loop vs. scores and reports only.

1.3 Core Philosophy

Position the platform as "AI Quality Infrastructure." One-off evaluation is like a physical exam whose report goes into a drawer — useless. It must embed into R&D workflows and run continuously. The platform is a general engine supporting all AI evaluation needs — unify scoring standards, accumulate data assets, make AI quality continuously trackable, not a pre-launch cram session.

2. Platform Architecture

Architecture serves one purpose: let the platform handle "all evaluation scenarios" without collapsing. Challenges from Chapter 1: scenario shapes vary wildly — code-gen evaluates generated code quality, knowledge-base cross-references docs and source, agent evaluation calls the live agent in real time. Writing a dedicated pipeline per scenario returns to "reinventing wheels." Solution: define domain-agnostic core abstractions, then split evaluation execution and analysis into two independent task chains.

2.1 Four Core Abstractions

After analyzing all scenarios, four domain-agnostic core entities emerged: Experiment , Dataset , Evaluator , EvalTask . The evaluation process always reduces to: take a dataset, judge AI output with some method. Dataset defines "what data," Evaluator defines "how to judge," AI Application defines "who to evaluate," Experiment binds the three, Task runs it.

Design highlights:

Core engine knows nothing about the test object. Experiment, Dataset, Evaluator, EvalTask are domain-agnostic. Domain-specific logic injects via evaluator types and scenario modules. Adding a new scenario adds zero lines to the core engine.

Evaluation and analysis split into two independent async tasks. EvalTask runs evaluations; AnalysisTask handles cross-experiment comparison, confidence calculation, and other heavy aggregation. Separation avoids re-analysis blocking evaluation callbacks and causing timeouts; evaluation and analysis are distinct concerns.

2.2 System Architecture Overview

The system splits into Platform Scheduling Layer and Evaluation Execution Layer .

Platform Scheduling Layer (Evaluation Scheduling App) handles "what to evaluate, when, and how to manage results": task-provider: ingests evaluation requests from frontend REST/HSF, SchedulerX scheduled jobs, and SDK integrations. task-biz: orchestration core — manages experiments/datasets/evaluators, evaluator factory dispatch, result aggregation, and confidence calculation. task-infrastructure: persists tasks/results and stores reports in OSS.

Evaluation Execution Layer has two channels by test-object type:

Synchronous channel: Agent-type scenarios call the tested AI application via HSF/HTTP to get output, then LLM judge scores and returns to platform.

Async Sandbox channel: Code-gen / knowledge-base / skill scenarios dispatch via MetaQ; eval-scenario modules consume and launch Skill sandboxes to run evaluations (evaluation logic is a black box to platform), then HTTP callback to platform for aggregation and persistence.

2.3 Evaluation Scenarios

155 evaluation scenarios on the platform group into four categories, each served by a pluggable eval-scenario module:

Code Generation (CODE_GENERATION) — evaluates AI-generated code quality; example: aaic code-gen evaluation.

LLM (Agent) — synchronous call to tested agent, evaluate output quality; example: idealab QA agent evaluation.

Knowledge Base Quality (KB_QUALITY) — cross-compare knowledge-base docs with source code; example: aaic knowledge-base evaluation.

Skill (SKILL) — evaluate Skill execution quality; example: data-construction skill evaluation.

This is "pluggable" in practice: adding a new scenario category means adding one scenario module; core engine unchanged, other scenarios unaffected.

3. Evaluator System: How to Judge Good vs. Bad

Evaluators are the platform's core component — they answer the fundamental question: "How to judge AI output quality?" Chapter 1 highlighted two evaluator challenges: judgment methods vary wildly (some call external scoring services, some run custom scoring code, some clone repos and read source line by line — a single evaluator shape cannot cover all); and LLM judges are unreliable (MT-Bench proved bias and jitter exist). This chapter answers both.

3.1 Diverse Judgment Methods → Four Pluggable Evaluator Types

The platform provides four evaluator types from light to heavy, generic to specialized:

MANUAL (Human): Humans score case-by-case on platform UI; total score = weighted sum of sub-dimension scores. Used for subjective quality review, new-scenario cold-start, and as calibration baseline for LLM judges. Highest cost; registered via platform UI.

CUSTOM (Custom): Developers write scoring logic in their own app, annotate with @Evaluator; SDK auto-registers as HSF service; platform calls remotely for scores. Code stays in-place, data never leaves the app. Low cost; auto-registration via annotation.

HTTP (External Service): Lightest integration for teams with existing scoring services. Configure service URL, request template (supports ${input} placeholders), and response score-extraction path in UI — no code. Low cost; platform UI configuration.

SKILL (Sandbox Skill): Heaviest type — runs a full evaluation Agent in a cloud sandbox. Suits complex evaluations that a single HTTP call cannot handle. Example: knowledge-base evaluation clones multiple Git repos, reads docs and source per manifest, cross-compares, aggregates scores. Knowledge-base, code-gen, and other heavy scenarios run here. High cost; registered via Diamond config with Skill name + version.

All four types are equal and composable in an experiment — one experiment can attach multiple evaluators, each scoring a dimension, then weighted into a total score (see Chapter 5). Adding a new judgment method means registering one more evaluator; the evaluation flow itself never changes. This directly answers "evaluator reinvention."

3.2 Unreliable Judges → Confidence Scoring

Using LLMs to judge AI carries the risk that "the judge itself is unreliable" — it jitters (same output, different scores) and biases (length, wording, etc.). MT-Bench/Chatbot Arena research summarized systematic biases: position bias (judges prefer first answer in pairwise), verbosity bias (prefer longer answers; repeated-list attack inflates scores), self-enhancement bias (tend to score own/family model outputs higher — limited evidence), and math/reasoning scoring failures (judges misled even on solvable problems).

Since bias is unavoidable, the platform makes judge jitter explicit. After all runs of an experiment complete,

ConfidenceScoreCalculator</strong> triggers consistency computation, using a confidence formula to expose judge jitter and discard outlier scores from single "glitch" runs.</p><p>Confidence measures "volatility itself" — based on overall standard deviation of all successful scores:</p><pre><code>confidence = max(0, 10 − stdDev × 5)

The final score excludes untrustworthy scores: it removes the single score with the largest deviation from the mean of the remaining scores, then averages the rest. Logic is deliberately conservative: we don't ask "which score is correct?" (unknowable); we only do one safe thing — remove the most discordant score so the rest agree. Result: final score isn't skewed by outliers, while confidence honestly exposes "how much this score set jitters." Confidence appears alongside scores in reports: high score but low confidence signals the consumer to decide whether to trust or request human re-review. This turns the industry's methodological call for "judge calibration" into a calculable, presentable engineering mechanism.

4. Dataset Management: What to Test With

Datasets answer the other half of evaluation — "what to test with." They address Chapter 1's dataset chaos (scattered, version mess) and the resulting hazard: if the benchmark itself is distorted, regression comparison loses meaning. Solution: converge dataset management into a unified service, plus a "production → precipitation" closed loop.

4.1 Unified Dataset Service

Replaces "each team maintains their own Excel" chaos:

Multi-source import: Excel file upload, ODPS table import, system data sync.

Dataset management: Beyond single datasets, introduces Dataset Group . Experiments bind to a dataset group (mutually exclusive with single dataset binding). Group holds a current_eval_dataset_id pointer to the latest version; each experiment run resolves the pointer to auto-use the newest data. Supports maintaining different-dimension datasets (normal, abnormal, edge cases) and binding multiple datasets from one group to an experiment. Version and grouping chaos converges to one pointer, one group.

Golden Dataset: Used in code-gen and data-construction scenarios to compare against tested output, continuously tracking whether the tested object's capability is stable. A crooked benchmark distorts stability judgment. Golden sets aren't built once: platform supports promoting typical data from daily evaluations into golden sets — promotion has permission checks; promoted data is marked. Benchmarks thicken as evaluations accumulate, making regression runs ever more accurate.

4.2 Data Recommendation & Acceptance

Beyond import and promotion, datasets have a third source: the platform's standalone "Data Recommendation & Acceptance" feature. Data-construction and similar scenarios batch-produce candidate data; after cleansing and scoring, entries enter a recommendation list. Scored entries can be manually accepted — users review case-by-case, accept, and entries auto-write into scene-grouped golden datasets. The loop: production → cleansing → scoring → acceptance → precipitation as benchmark — closes inside the platform. It solves "benchmark accumulation is slow": golden sets don't wait for one-off manual curation; high-quality data from daily evaluations flows continuously into the benchmark library.

5. Experiment Management: How to Run & Use

If evaluators and datasets solve "how to judge" and "what to test with," experiments solve "how to run." They are the scheduling hub: bind test scenario, dataset, evaluators, define "who to evaluate, with what data, how to judge." Challenges: high experiment creation cost, hard to reuse standards; black-box drift needs continuous patrol, can't rely on humans to remember to trigger; single scores untrustworthy, need to separate volatility sources; results don't drive decisions, issues must surface structurally. This chapter covers the full experiment lifecycle: creation (with template reuse), triggering (manual/scheduled/auto-on-release), execution (async, large-repo parallelism), and two-layer stability measurement after multi-run.

5.1 Creating Experiments

Creating an experiment answers three questions:

Who to evaluate (select test scenario)

What data to use (select dataset(s), support multi-select, or bind dataset group)

How to evaluate (select evaluator(s))

Wizard-style creation: three steps, done. High-frequency scenarios can be saved as scheduled experiment templates (default configs for dataset, evaluators, scenario). New experiments instantiate from template on schedule, inheriting all configs. Templates solve "standard reuse" — same evaluation standard doesn't need re-assembly each time. Beyond scenario/dataset/evaluator selection, experiments configure a total-score formula: each evaluator's sub-score weight defined by the user; stability evaluators have separate weights. Total = Σ(evaluator_sub_score × weight) + Σ(stability_evaluator_score × weight) . Significance: same evaluator, different experiments/scenarios can weight differently per their focus — raise rule-based weight for compilation-pass priority, raise LLM-judge weight for semantic quality — without changing evaluation logic for a single standard.

5.2 Triggering Experiments

Three ways to run a created experiment:

Manual trigger: For integration debugging or immediate conclusions; click on UI to run.

Scheduled trigger: Periodic auto-run, e.g., daily 08:30 patrol. This catches the "black-box behavior drift" from Chapter 1 — local degradation doesn't knock on its own door.

Change trigger: In knowledge-base evaluation, new repo commits are scanned on schedule and auto-trigger evaluation; newly registered business domains trigger immediately. Test object changes → re-evaluate, no human memory required.

5.3 Evaluation Consistency

Looking at one score, you can't tell whether volatility comes from the tested scenario's unstable output or the evaluator's scoring jitter. Platform measures these separately: evaluator side via confidence (Chapter 3); test-scenario side via consistency — same scenario, same dataset, multi-run, are results stable? Typical situation: one run scores 8.5, next day 6.2. Did the tested object degrade or did the evaluator mis-score? Two separate metrics answer. Consistency evaluation can be triggered independently: platform provides a dedicated consistency evaluation task — specify experiment and dataset. Each experiment run persists as an execution record: score, dimension breakdown, structured report, issue list all attached to that run; multi-run comparison enabled.

5.4 Experiment Result Analysis

One experiment runs hundreds of cases; score output is just the start. Which cases to inspect first? Platform auto-triggers analysis task after evaluation, categorizing data by score: Perfect, Normal Low, Critical Low (dynamically extensible). Low scores auto-cluster; bad cases no longer require manual sifting. Deep-dive on a single case supported. Special analysis form: Issue Mining — only finds issues, no scoring, not limited to a single experiment. Runs on the same analysis task chain; can link to a specific experiment or run independently. Covers four sources: incremental requirement issues, knowledge-base issues, data-construction skill issues, and rules extracted from issues. One mining run yields dozens to hundreds of issues; flat listing buries critical problems in minor nitpicks. Every issue carries dimension and severity: CRITICAL (severe), MAJOR (moderate), MINOR (minor), judged by impact. Run records accumulate counts per severity — what to fix first is obvious. Issue list filterable by dimension, severity, disposition status. Confirm real issue → accept; false positive → reject with reason. Disposition states ( Pending / Accepted / Rejected ) persist — fixes traceable, false positives auditable, progress visible on run records. Evaluation results become an actionable, trackable issue list, not just a score — directly answering "results don't drive decisions."

6. Multi-Scenario Production Practices

Architecture must be validated by real scenarios. This chapter walks one landed evaluation per category: agent evaluation, knowledge-base evaluation (overall + retrieval), code-gen evaluation, skill evaluation.

6.1 Agent Evaluation: idealab QA Agent

Scenario: Evaluate idealab QA agent answer quality — user asks, agent answers, evaluate accuracy and problem resolution. Tested type: Agent type; synchronous HTTP call to live agent for answer; real-time evaluation requiring live test scene. Evaluator types: Manual evaluator (subjective review) or HTTP evaluator (platform provides generic QA evaluator), three sub-dimensions scored by LLM judge via Judge Prompt.

6.2 Knowledge-Base Evaluation: aaic Code-Gen Knowledge Base (Overall + Retrieval)

Two parts: Overall evaluation — knowledge-base doc quality; Retrieval evaluation — quality when knowledge is actually retrieved and used. Overall Evaluation

Scenario: Assess KB doc accuracy, structural completeness, compliance.

Evaluator: SKILL (DIRECT mode) — clones three repos (KB + Workspace + Source), three-stage progressive validation: Structure Complete → Format Compliant → Content Correct ; each layer independent check, independent score and report.

Evaluation method: KB organized in layers G1–G7; evaluation runs per layer; each layer produces independent score, report, and verdict ( PASS / CONDITIONAL / FAIL); all layer reports viewable and downloadable.

Layered design directly targets Chapter 1's "local degradation" problem: each layer's report stands alone; any layer's regression isn't masked by other layers' aggregate score.

Retrieval Evaluation Writing good KB is only half the battle — whether knowledge is actually retrieved and used correctly during code-gen is the ultimate test. Code-gen evaluation produces two knowledge-dimension scores: knowledgeRecall (were required knowledge pieces retrieved?) and knowledgePrecision (are retrieved pieces correct?), measuring KB retrieval quality from the code-gen pipeline backward.

6.3 Code-Gen Evaluation: aaic Code-Gen Evaluation

Covered in a separate deep-dive article; not expanded here.

6.4 Skill Evaluation: Data-Construction Skill

Scenario: Evaluate AI data-construction Skill capability — given natural-language instruction like "construct a normal user registration data," can the Skill produce correct business-compliant test data? Evaluator: SKILL type — evaluation Skill directly uses live data-construction Skill's real output, scores per evaluation rules. Cleansing & Scoring: Batch-produced candidate data cleansed to canonical form, then scoring assigns quality scores; unscored entries have scheduled catch-up runs. Recommendation: Cleansed candidates enter recommendation list, browsable by day; users accept data needing evaluation. Golden Dataset: Evaluation results support manual promotion to golden data; once written to golden set, becomes regression benchmark for that Skill — fixed questions, repeated comparison, capability regression visible at a glance. Issue Analysis: Post-evaluation auto-triggers issue analysis, summarizes root causes and suggests fixes per data-construction command dimension.

7. Summary & Future Directions

7.1 Summary

Building this platform changed our understanding of "evaluation." Initially thought evaluation = scoring; later realized score alone is insufficient: an 8.5 means nothing unless you know it's stable and the judge is reliable — only then dare you decide with it. What evaluation truly precipitates is standards: datasets precipitate "what to test," evaluators precipitate "how to define good." Standards established, "what is a good AI application" becomes definable. Current status: platform onboarded four test-object categories (code-gen, agent, knowledge-base, skill), 155 evaluation scenarios running; four pluggable evaluator types cover from "call an API" to "clone three repos, per-node compare"; datasets converged from scattered Excels to unified service + dataset groups + golden sets + recommendation-acceptance loop; execution moved from "human remembers to run" to manual/scheduled/change-triggered continuous patrol; results evolved from a single score to dual metrics (consistency + confidence), structured issue lists, three-level severity. Example: knowledge-base evaluation compressed single-domain review from days to 20 minutes, 200+ nodes full coverage, thousands of "doc vs. source" comparisons automated — all repeatable, comparable, traceable. This meets Chapter 1's goal: turn the feeling of "good/bad" into trustworthy, comparable, issue-revealing scores. What the platform does today: makes scores trustworthy, surfaces issues. What it doesn't yet do: real risk-gating gates, self-evolving evaluation logic — honestly listed in future directions. Ultimately, evaluation is a dashboard for AI application iteration: dashboard doesn't drive the car, but without it you don't know how fast you're going or how far you can go. Hope this platform lets more teams drive AI applications faster and better with confidence.

7.2 Future Directions

Direction 1: Test-scene gates & optimization loop. Today results consumed via reports, trends, pushes — action still human-dependent. Next: make evaluation a real gate — block on sub-standard scores or critical issues, not just alert. Coupled with optimization loop & monitoring: continuously track each scene's score & issue trends, detect degradation early, locate, verify fix, make "evaluate → improve → re-evaluate" routine.

Direction 2: Self-evolving evaluation logic. Today evaluation logic relies on humans to set standards and fix. Vision: system discovers its own evaluation logic issues — e.g., cross-domain statistical anomalies from judge behavior: when most domains show same misjudgment on same node type, can flag judge rule problematic without knowing "ground truth." Build "analysis → optimization suggestion → human review → fix" loop; AI proposes, human approves. Still in design/exploration, not live.

Direction 3: Lower entry barrier & more flexible usage. Make new scenarios easier to onboard — templates & best practices, reuse what's reusable; make usage freer — evaluators, datasets, score formulas as building blocks users combine at their own pace and standards, not forced into a fixed mold.