Redefining Agent Evaluation: An Offline LLM-as-a-Judge Framework and Practice

Background: Challenges and Needs in Agent Evaluation

AI outbound calls have evolved from simple FAQ bots to complex AI agents that must handle both knowledge‑answering (FAQ) and procedural execution (SOP) with autonomous planning and tool invocation. As agent capabilities grow, continuous technical iteration—such as prompt tuning, RAG knowledge‑base expansion, or base model replacement—requires a quantifiable metric system to prevent regression and to measure optimization gains.

Regression prevention: Ensure that local optimizations or knowledge‑base updates do not degrade other core abilities (avoid fixing 20 bad cases while breaking 80 good cases).

Optimization quantification: Provide clear data‑driven evidence that a new model or deployment improves over the baseline.

Why Traditional LLM Evaluation Falls Short

Conventional LLM benchmarks focus on static correctness, fluency, and compliance of a single output. Agent evaluation must also verify the "intermediate process" —whether the agent triggers the right tool at the right time and passes correct parameters.

In complex outbound scenarios, agents face unpredictable multi‑turn dialogues, making traditional testing inadequate.

Industry Research on LLM‑as‑a‑Judge

We surveyed major industry approaches. For the large‑vehicle AI outbound use case, evaluation targets are split into base‑model (LLM) optimization and Harness (non‑LLM) optimization. The core definition is: Agent = LLM + Harness LLM handles textual understanding and intent detection, while Harness provides system prompts, RAG retrieval, and tool‑execution capabilities.

Evaluation Scenarios and Methods

Evaluation Scenario | Evaluation Object | Evaluation Method | Core Mechanism | Specific Scheme
--- | --- | --- | --- | ---
Base‑model (LLM) optimization | Agent final result | Reference‑free / Point‑wise | Rubric without reference; judge scores directly (1‑5 or Yes/No) | –
Base‑model (LLM) optimization | Agent final result | Reference‑based | Golden answer alignment; judge compares output to reference facts | Google Gemini benchmark (GPQA, MATH, etc.)
Base‑model (LLM) optimization | Agent final result | Pairwise / GSB | Judge receives A and B outputs, blind‑compares, outputs Win/Tie/Loss | Google Auto SxS side‑by‑side
Harness (non‑LLM) optimization | Agent final result | Reference‑free / Point‑wise | Same rubric as above | Anthropic safety rules
Harness (non‑LLM) optimization | Agent final result | Reference‑based | Golden answer alignment | Anthropic regression testing
Harness (non‑LLM) optimization | Agent final result | Pairwise / GSB | Same blind‑compare mechanism | –
Agent intermediate process | Trajectory evaluation | Agent‑as‑a‑Judge | Judge scores whole dialogue trajectory and business outcome | –
Agent intermediate process | Trajectory evaluation | Agent‑as‑a‑Judge | Judge audits API call chain line‑by‑line in sandbox | –

Design: Customized Hybrid Scheme

Existing methods alone cannot meet our needs:

Reference‑based only: Determines correctness but cannot rank “better” among correct answers.

GSB only: Ranks quality but cannot guarantee factual correctness.

We therefore combine them:

Stage 1 – Reference‑based (regression guard): Verify that candidate models do not introduce regressions on good cases.

Stage 2 – GSB (fine‑grained optimization): After correctness is confirmed, compare subjective dimensions to select the superior model.

Implementation Details

Steps and scoring logic are summarized in the table below:

Step | Evaluation Metric | Scoring Logic
--- | --- | ---
1 – Reference‑based | Semantic consistency rate before/after iteration | Count percentage of outputs that regress
2 – GSB | Win rate (new model vs baseline) | Use GSB (Good/Same/Bad) to compute win rate

Core prompt skeletons are illustrated in the following images:

Harness Optimization Practice

For Harness improvements, the focus shifts from “subjective preference” to “deterministic compliance”. The goal is to correct concrete bugs without altering existing SOP compliance. We therefore use a reference‑based scheme only.

Future work includes adding trajectory and log evaluation to capture the intermediate planning‑action loop.

Implementation and Results

During deployment we observed systematic bias in the judge model: it tended to favor the output placed first in pairwise tests. Literature ("Judging the Judges: A Systematic Study of Position Bias in LLM‑as‑a‑Judge") confirms this bias.

When swapping the positions of two outputs in GSB, the judge consistently chose the earlier one.

To mitigate bias we applied two strategies:

Judge model selection: Use a model whose capability far exceeds the evaluated agent. In our case, DeepSeek V3.2 was chosen to judge Qwen3‑235B, reducing self‑preference.

Systematic bias elimination: Engineer prompts and voting mechanisms to neutralize position bias.

Performance metrics:

Inference engine switch: 97% agreement with human labels.

Regression testing: 98% agreement.

Efficiency gains: Traditional manual offline evaluation processes ~100 cases/hour; the LLM‑based engine reduced 100 cases to under 5 minutes.

GSB pipeline (limited by DeepSeek concurrency) processes 100 cases in ~5 hours with zero human intervention.

Future Plans

Introduce trajectory and trace evaluation to precisely measure the intermediate process.

Automate assessment of the planning‑action loop in a simulated environment.

Build a closed‑loop "evaluation‑attribution‑repair" system that links evaluation results to root‑cause analysis and generates targeted prompt or strategy recommendations.

Author: AI Application Group (余嘉慧, 吴立薪, 万勇韬)