From Golden Metrics to Rubric: Building a Quantifiable, Explainable Evaluation Loop for AI Agents

The core problem after data ingestion is determining how well an agent performs . Because agent outputs are open‑ended, manual spot‑checks are costly and cannot form reusable standards. AgentLoop solves this by establishing a quantifiable, explainable, reusable evaluation pipeline that turns “good or bad” into weighted scores and traceable evidence.

Two‑Layer Evaluation Architecture

Evaluator : defines what to evaluate (e.g., task completion), its type (Agent or Code), output schema, and a Rubric that maps each metric to score buckets and weights. The Rubric is independent of data and can be reused.

Evaluation Task : selects which evaluators to apply, which data to run on, and when to run. It is analogous to a test paper (the evaluator) and an exam session (the task).

Evaluator Types

Code evaluation : rule‑based scoring (e.g., format, length) – cheap but limited to expressible criteria.

Agent evaluation : a dedicated evaluation agent reads input, output, and execution trace, applying the Rubric to assess semantic correctness and process quality – more accurate but costlier.

Defining Golden Metrics and Generating a Rubric

For a customer‑service agent, two golden metric families are identified:

Answer quality (correctness, relevance).

Execution process (tool calls, reasoning path).

Instead of hand‑crafting the Rubric, the author prompts ChatGPT with a request to produce a Rubric that lists each metric, its scoring conditions, weight, and a total‑score formula. The LLM returns a complete Rubric covering both metric families, including suggestions such as a hard gate and evidence coverage.

Creating the Evaluator

The Rubric is placed into the evaluator’s Prompt. Using AgentLoop’s built‑in Skill, the prompt is generated and then copied into the console when creating a new evaluator of type Custom Agent . The prompt includes the Rubric and references to three variables: input (user query), output (agent answer), and trace.agent (the full reasoning trace).

The evaluator’s output schema defines fields such as score, raw-weighted score, final score, decision, scenario type, summary, and explanation. A rubric version field is also added to track rubric evolution.

Configuring the Evaluation Task

Because the goal is to assess answer quality and process, the task uses Trajectory data (the agent’s reasoning trace) rather than raw service‑call logs. Two execution strategies are offered:

Continuous evaluation : run on each new record in real time, suitable for online quality monitoring.

Historical evaluation : run once over a bounded time window, useful for post‑mortem analysis.

Sampling configuration limits cost; the demo sets a maximum of 100 samples with a 100 % sampling ratio, allowing a quick sanity run before scaling.

Field Mapping

The platform’s data fields are mapped to the evaluator’s variables ( trace.input → input, trace.output → output, trace.agent → trace.agent) so the evaluator receives the correct inputs.

Running the Evaluation and Analyzing Results

After execution, the result page shows per‑record input/output on the left and evaluation outcomes on the right, including average scores, weighted scores, decision, and detailed evidence. Because scores can fluctuate, the author recommends averaging over multiple runs for stability.

Low‑scoring cases are fed back into a “badcase” dataset, forming the ammunition for the next round of offline experiments. This closes the loop: evaluation → badcase collection → targeted re‑training or prompt refinement → re‑evaluation.

Key Takeaway

Evaluation transforms expert notions of “good” into machine‑executable, interpretable assets, providing a common measurement foundation for downstream experimentation and continuous improvement.