Tagged articles

LLM-as-a-Judge

8 articles · Page 1 of 1

Nov 6, 2026 · Artificial Intelligence

Langfuse Three-Level Agent Evaluation: Outcome, Trajectory & Step Framework Principles & Implementation

This article details Langfuse's three-level evaluation framework for AI agents—Outcome, Trajectory, and Step—explaining how its Trace, Evaluator, Dataset, and Experiment components enable decoupled observability and evaluation, with concrete implementation patterns for code-based rules, LLM-as-a-Judge, and human annotation to build reliable CI/CD gates and production monitoring loops.

AI TestingAgent EvaluationCI/CD Gates

0 likes · 33 min read

Langfuse Three-Level Agent Evaluation: Outcome, Trajectory & Step Framework Principles & Implementation

DataFunTalk

Nov 3, 2026 · Artificial Intelligence

AWS Skill Eval: Making Agent Skills Testable, Traceable, and CI-Ready

AWS introduces skill-level evaluation in Strands Evals and Bedrock AgentCore, adding three-layer checks — invocation, selection accuracy, and instruction following — that move agent evaluation from final answers to execution traces, integrate with CI/CD pipelines, and treat skills as versioned, testable engineering assets.

AWSAgent SkillsBedrock AgentCore

0 likes · 15 min read

AWS Skill Eval: Making Agent Skills Testable, Traceable, and CI-Ready

DataFunTalk

Oct 20, 2026 · Artificial Intelligence

AWS Skill Eval Turns Agent Skills into Testable Engineering Assets

AWS introduces skill-level evaluation in Strands Evals and Bedrock AgentCore, enabling three-layer checks — invocation, selection accuracy, and instruction following — that treat SKILL.md files as auditable engineering components integrated into CI/CD pipelines, shifting agent evaluation from final answers to execution traces.

AWSAgent SkillsBedrock AgentCore

0 likes · 16 min read

AWS Skill Eval Turns Agent Skills into Testable Engineering Assets

Linyb Geek Road

Sep 24, 2026 · Artificial Intelligence

How Jev Cuts Agent Orchestration Costs 80% with Dynamic Harness Generation

Jev, a specialized classification model, reduces agent orchestration costs by 80% by handling routing, evaluation, intent decomposition, and dynamic harness generation at a fraction of GPT-4o's cost, enabling self-orchestrating multi-agent systems while maintaining quality, though plan-level error detection remains an open challenge.

Cost OptimizationJevLLM routing

0 likes · 12 min read

How Jev Cuts Agent Orchestration Costs 80% with Dynamic Harness Generation

Data Bricklaying Diary

Aug 26, 2026 · Artificial Intelligence

AI Evaluation Sets ≠ Training Holdouts: Golden Data, Hard Cases, Contamination & Regression

This article explains why enterprise AI evaluation requires purpose-built datasets — golden baselines, challenge cases, red-team tests, regression suites, and online replays — designed from capability questions, risk lists, and real failures, with structured answers, contamination governance, tiered LLM judging, and version-locked regression pipelines gated by risk-based release thresholds.

AI evaluationLLM-as-a-JudgeRegression Testing

0 likes · 24 min read

AI Evaluation Sets ≠ Training Holdouts: Golden Data, Hard Cases, Contamination & Regression

Woodpecker Software Testing

Aug 23, 2026 · Artificial Intelligence

How to Choose the Right Model Evaluation Method – From Exact Match to LLM-as-a-Judge

This guide explains objective metrics such as Exact and Fuzzy Match, the QUEST framework for human evaluation, rubric design and calibration, the LLM-as-a-Judge approach with its biases and trade‑offs, and a five‑dimensional evaluation framework for building robust, explainable and fair AI systems.

AI assessmentLLM-as-a-JudgeRubric

0 likes · 26 min read

How to Choose the Right Model Evaluation Method – From Exact Match to LLM-as-a-Judge

Huolala Tech

Aug 12, 2026 · Artificial Intelligence

Redefining Agent Evaluation: An Offline LLM-as-a-Judge Framework and Practice

The article presents a quantitative offline evaluation system for AI agents in real outbound-call scenarios, combining reference‑based scoring with pairwise GSB methods, addressing regression and optimization, mitigating systematic bias through judge model selection and majority‑vote adjustments, and delivering an objective, high‑efficiency benchmark.

Agent EvaluationGSBLLM-as-a-Judge

0 likes · 16 min read

Redefining Agent Evaluation: An Offline LLM-as-a-Judge Framework and Practice

AI Engineer Programming

Jun 11, 2026 · Artificial Intelligence

How to Build Truly Effective LLM-as-a-Judge Evaluators

The article explains how to construct reliable LLM-as-a-Judge evaluators by combining deterministic code checks for syntactic validation, designing clear semantic evaluation rubrics, choosing appropriate output formats, calibrating with human‑labeled data, mitigating known model biases, and integrating trace‑based monitoring into production workflows.

AI safetyLLM EvaluationLLM-as-a-Judge

0 likes · 15 min read

How to Build Truly Effective LLM-as-a-Judge Evaluators