How to Build Truly Effective LLM-as-a-Judge Evaluators

Core Points

Human review cannot scale, so LLM‑as‑a‑Judge is needed for batch evaluation of LLM applications and agents.

Use Code When You Can

Prefer deterministic checks for aspects that can be automated: JSON validation, known ID verification, and trace inspection. Use LLM judges only for semantic judgments such as whether the answer truly solves the user problem, is grounded in retrieved context, respects safety constraints, and selects appropriate tools.

import json
from jsonschema import ValidationError, validate

TOOL_CALL_SCHEMA = {
    "type": "object",
    "required": ["tool_name", "arguments"],
    "properties": {
        "tool_name": {"type": "string"},
        "arguments": {"type": "object"}
    }
}

ALLOWED_TOOLS = {"lookup_customer_profile", "refund_order"}

def valid_tool_call(output: str) -> bool:
    try:
        payload = json.loads(output)
        validate(payload, TOOL_CALL_SCHEMA)
        return payload["tool_name"] in ALLOWED_TOOLS
    except (json.JSONDecodeError, ValidationError, KeyError):
        return False

Practical rule: if the answer can be checked without interpretation, use code; otherwise use a judge.

Design Evaluation Criteria Before Prompting

Many judges fail because the rubric is ill‑defined. A reliable rubric contains five parts: evaluation goal, available inputs, allowed labels, decision rules, and examples.

评估目标：智能体是否解决了用户的支持请求？

可用标签：
- resolved：用户收到了正确、可执行的答案，且有必要的工具证据支撑。
- partially_resolved：智能体有进展，但还差一个必要步骤。
- unresolved：智能体未能回答、给了错误指导，或缺乏必要证据。
- insufficient_evidence：trace 中缺乏足够证据来评定任务完成情况。

决策规则：
- 用户不得不重复同一请求，不标为 resolved。
- 没有工具证据，不标为 resolved。
- 答案看似合理但无工具结果支撑，标为 unresolved。
- 关键工具结果缺失，标为 insufficient_evidence，而非猜测。
- 升级质量单独评估，不影响本项标签。

This rubric measures only the task‑completion dimension.

Choose an Output Format That Matches the Decision

Four common label types are used:

Boolean labels – ideal for gate‑keeping (e.g., valid/invalid). Add insufficient_evidence or needs_review when evidence is missing.

Categorical labels – useful when there are several distinct states.

Ordinal labels – require clearly anchored tiers; avoid mixing unrelated properties.

Open numeric scores – attractive but easy to misuse; only use when a calibrated continuous scale exists.

Run Evaluators Near the Trace

Labels are useful only when the execution trace is available. The workflow is:

Extract representative samples from production or pre‑production traces.

Label these samples with the team’s actual tags.

Write a fixed rubric with labels and decision rules.

Run the judge on the labeled set and review inconsistencies in Phoenix.

Refine the rubric or add samples, record results back to the trace and dataset.

Once the evaluation pipeline runs, the loop becomes: filter failing samples → inspect trace and judge explanation → categorize root cause → add representative failures to the dataset → fix → re‑run → verify no new failures.

Stabilize the Rubric Before Picking a Model

The strongest model is not always the best judge. Large, cutting‑edge models may have higher latency and cost, while smaller models often suffice for clear boolean labels. Test multiple model families to reduce self‑preference bias.

Explain, But Don’t Treat Explanations as Truth

Explanations help debugging but are not evidence. Output contracts should separate fields:

{
  "label": "unsupported",
  "explanation": "答案说退款将在 24 小时内到账，但政策上下文只写明通常在 5 个工作日内处理。",
  "evidence": ["退款通常在 5 个工作日内处理"]
}

For complex checks, require the judge to point out the exact unsupported statement or missing tool call.

Mitigate Known Biases

Positional bias – randomize answer order.

Length bias – constrain scoring scales.

Self‑preference bias – use judges from different model families.

Authority bias – require explicit evidence fields.

Evaluation drift – version‑manage rubrics and run canary test sets.

Hallucination – enforce structured evidence.

Treat Judges Like Production Code

Apply version control, automated testing, and monitoring to judges just as you would to any production component.

Evaluate the Trace, Not Just the Final Answer

Assess multiple layers:

Final answer quality : correctness, grounding, completeness.

Tool choice : was the right tool selected? any unnecessary calls?

Tool parameters : are they specific and valid?

Tool result handling : correctly interpreted? retries handled?

Trace efficiency : shortest reasonable path vs. redundant loops.

Conversation outcome : did the user achieve their goal?

Validate Judges with Human Calibration

The first version of a judge is a hypothesis. Build a small validation set containing clear passes, clear failures, and historically disputed cases. Measure accuracy, precision/recall, F1, Cohen’s kappa, and error breakdowns by domain, prompt version, and user segment.

Example: 100 annotated support chats – 55 resolved, 20 partially resolved, 15 unresolved, 10 insufficient evidence. The initial judge agreed with humans on 82 cases; the 18 mismatches revealed missing tool evidence, ambiguous upgrade handling, and absent traces.

Design for Known Biases

Apply the mitigation strategies listed above throughout the rubric and model‑selection process.

Use the Judge for Its Intended Tasks

Deployment gate‑keeping – tune based on actual triggered actions.

Monitoring – stable trend detection; occasional label errors are tolerable.

Dataset curation – surface samples worth human review.

Prompt iteration – reliable pairwise comparison to detect genuine improvements.

Conclusion

The most robust system combines deterministic code checks, semantic LLM judges, human calibration, and trace observability. This turns evaluation from a single dashboard number into a feedback loop: observe behavior → measure failure → fix system → verify the fix.