Tagged articles

LLM-as-a-Judge

4 articles · Page 1 of 1

Aug 26, 2026 · Artificial Intelligence

AI Evaluation Sets ≠ Training Holdouts: Golden Data, Hard Cases, Contamination & Regression

This article explains why enterprise AI evaluation requires purpose-built datasets — golden baselines, challenge cases, red-team tests, regression suites, and online replays — designed from capability questions, risk lists, and real failures, with structured answers, contamination governance, tiered LLM judging, and version-locked regression pipelines gated by risk-based release thresholds.

AI evaluationLLM-as-a-Judgechallenge set

0 likes · 24 min read

AI Evaluation Sets ≠ Training Holdouts: Golden Data, Hard Cases, Contamination & Regression

Woodpecker Software Testing

Aug 23, 2026 · Artificial Intelligence

How to Choose the Right Model Evaluation Method – From Exact Match to LLM-as-a-Judge

This guide explains objective metrics such as Exact and Fuzzy Match, the QUEST framework for human evaluation, rubric design and calibration, the LLM-as-a-Judge approach with its biases and trade‑offs, and a five‑dimensional evaluation framework for building robust, explainable and fair AI systems.

AI assessmentLLM-as-a-JudgeRubric

0 likes · 26 min read

How to Choose the Right Model Evaluation Method – From Exact Match to LLM-as-a-Judge

Huolala Tech

Aug 12, 2026 · Artificial Intelligence

Redefining Agent Evaluation: An Offline LLM-as-a-Judge Framework and Practice

The article presents a quantitative offline evaluation system for AI agents in real outbound-call scenarios, combining reference‑based scoring with pairwise GSB methods, addressing regression and optimization, mitigating systematic bias through judge model selection and majority‑vote adjustments, and delivering an objective, high‑efficiency benchmark.

Bias MitigationGSBLLM-as-a-Judge

0 likes · 16 min read

Redefining Agent Evaluation: An Offline LLM-as-a-Judge Framework and Practice

AI Engineer Programming

Jun 11, 2026 · Artificial Intelligence

How to Build Truly Effective LLM-as-a-Judge Evaluators

The article explains how to construct reliable LLM-as-a-Judge evaluators by combining deterministic code checks for syntactic validation, designing clear semantic evaluation rubrics, choosing appropriate output formats, calibrating with human‑labeled data, mitigating known model biases, and integrating trace‑based monitoring into production workflows.

AI SafetyLLM EvaluationLLM-as-a-Judge

0 likes · 15 min read

How to Build Truly Effective LLM-as-a-Judge Evaluators