Tagged articles
Agent Evaluation
47 articles · Page 1 of 1

ArchitectNov 16, 2026 · Artificial Intelligence
The article deconstructs Anthropic's Daily Brief example to show how effective agent automation requires cursor-based progress tracking, pre-send verification via ledgers, idempotent delivery confirmation, strict permission boundaries, budget guards, and evaluation against human-labeled ground truth to ensure automation truly reduces workload rather than shifting it to verification.
Agent AutomationAgent EvaluationAnthropic
0 likes · 38 min read

This article explains AI Agent fundamentals using a personal assistant analogy, then details how to design evaluation cases, interpret results across quality, stability, and cost dimensions, compare versions, and use evaluation data to make informed release decisions, illustrated with a home renovation Agent case study.
AI agentAgent EvaluationEvaluation Design
0 likes · 21 min read

Einsia AI's WorkflowArena benchmark evaluates 9 frontier models across 145 long-horizon tasks in 8 domains, revealing that even the best system (Claude Opus 5.5) achieves only 36.95% full solve rate, exposing multi-agent orchestration as a critical weakness, with distinct orchestration styles between Claude and GPT models, and showing that strong orchestration can significantly boost weaker executors.
AI agentsAgent EvaluationClaude Opus 5.5
0 likes · 17 min read

This comprehensive guide covers 10 essential AI Agent interview questions, detailing core concepts like Agent vs LLM, Workflow vs Agent architectures, reasoning paradigms (ReAct, CoT, Plan-and-Execute, Self-Refine), Function Calling, MCP, error handling, memory systems, context management, RAG pipelines, and evaluation methodologies with code examples and diagrams.
AI agentAgent EvaluationFunction Calling
0 likes · 39 min read

This article details Langfuse's three-level evaluation framework for AI agents—Outcome, Trajectory, and Step—explaining how its Trace, Evaluator, Dataset, and Experiment components enable decoupled observability and evaluation, with concrete implementation patterns for code-based rules, LLM-as-a-Judge, and human annotation to build reliable CI/CD gates and production monitoring loops.
AI TestingAgent EvaluationCI/CD Gates
0 likes · 33 min read

Alibaba's 1683 shares its production-grade Agent evaluation framework combining offline golden sets with online log sampling, using Task×Trial×Trace×Outcome decomposition, executable Rubrics with deterministic gates, LLM Judges constrained by evidence citations and rule-based scoring, and a closed loop that routes failures to targeted fixes and regression testing.
AI agentAgent EvaluationDual-Track Evaluation
0 likes · 20 min read

Tsinghua, Microsoft, and collaborators introduce FDE-Gym, a reproducible benchmark for AI delivery agents across 20 customer scenarios and 95 tasks, revealing that even Claude Opus 5.5 solves only 14 tasks at the most realistic level, with the core bottleneck being discovery of implicit constraints and translation of facts into deployed systems.
AI delivery agentsAgent EvaluationBenchmark
0 likes · 13 min read

This lesson teaches how to implement metacognition in AI agents — enabling self-assessment via confidence scores, self-consistency, multi-model voting, and internal knowledge checks — and covers a complete three-layer evaluation framework (task, trajectory, experience) with runnable code for production-grade evaluation harnesses, key metrics like pass@k vs pass^k, and major benchmarks including HumanEval, SWE-bench, GAIA, and WebArena.
AI agentsAgent EvaluationBenchmarks
0 likes · 22 min read

This article systematically breaks down six industrial-grade Agent evaluation metrics (PlanQuality, PlanAdherence, ToolCorrectness, ArgumentCorrectness, TaskCompletion, StepEfficiency) and four RAG baseline metrics (Faithfulness, Answer Relevancy, Context Precision, Context Recall), detailing their definitions, calculation formulas, underlying principles, and key differences in evaluation scope, failure modes, granularity, and judge alignment targets.
AI-evaluationAgent EvaluationDeepEval
0 likes · 22 min read


ArchitectOct 28, 2026 · Artificial Intelligence
Anthropic finds that splitting AI agents by business function often increases latency due to handoff overhead; they recommend consolidating sub-agents into a main loop using four patterns — merge, fold, as-tool, keep — with Skills for business rules, tools for actions, and task-level evaluation to catch regressions.
AI agentsAgent EvaluationAnthropic
0 likes · 24 min read

Alibaba Cloud details its methodology for constructing and maintaining high-quality evaluation datasets for the STAROps AIOps agent, covering fault injection, signal capture, evidence extraction, quality checks, environment construction, and continuous iteration to ensure reliable root cause analysis assessments.
AIOpsAgent EvaluationRoot Cause Analysis
0 likes · 42 min read

The article introduces EvoMemBench, a benchmark evaluating AI agent memory across four dimensions — task-internal knowledge, task-internal execution, cross-task knowledge, and cross-task execution — revealing five key findings: memory isn't always better than raw context, helps more on hard tasks, is critical for long-horizon execution, retrieval does not equal revision, and no single memory type fits all tasks. It concludes with five engineering principles for building effective memory systems.
AI agent memoryAgent EvaluationEvoMemBench
0 likes · 20 min read

ByteDance's TraceDance system automatically converts 252,557 real-world agent deployment trajectories into 107 behavioral benchmarks with 4,125 test instances, evaluating nine frontier models including Claude, GPT, and DeepSeek, which achieve only a 26.7% average pass rate on decision-point continuation tasks, revealing gaps between task completion and critical behavioral decisions.
Agent EvaluationBehavioral BenchmarksByteDance
0 likes · 17 min read

Anthropic argues that multi-subagent architectures often mirror organizational charts rather than technical needs, showing single-loop agents with progressive skill disclosure outperform complex pipelines in quality, latency, and cost, and provides a migration framework.
AI agentsAgent EvaluationAnthropic
0 likes · 22 min read

The article details a continuous delivery pipeline for high-quality datasets, combining Dataset CI for contract validation and Eval CI for task regression, with automated quality gates, manual approvals, canary releases, and full-baseline rollback, using a policy knowledge base as a running example.
Agent EvaluationContinuous DeliveryData Governance
0 likes · 29 min read

The article explains how successful Agent execution traces can still produce wrong answers, using a procurement case where an Agent cited an outdated contract attachment. It details a systematic approach: trace errors through observability across indexing, retrieval, context assembly, and generation; convert feedback into reproducible test samples with versioned inputs and expected behaviors; evaluate using rule-based checks, model-assisted judging, and human review; separate critical error categories instead of relying on a single score; maintain regression suites with both failure and success cases; and rerun tests in isolated environments while recording versions to distinguish new results from original incidents.
Agent EvaluationRegression TestingRetrieval-Augmented Generation
0 likes · 15 min read

AWS open-sourced Strands Harness, a pre-assembled agent runtime that decouples the model from the execution system; benchmarks show using the same Claude Fable 5 model, Strands achieves 77% lower cost and higher accuracy than Claude Code by optimizing context management, prompt caching, and layered memory, proving harness design critically impacts agent performance.
AWSAgent EvaluationAgent Runtime
0 likes · 18 min read

The article explains why a tool's success response doesn't guarantee an Agent task is done, detailing three verification layers (call, execution, task), upfront acceptance criteria, timeout recovery via state reconciliation, and evaluation metrics that separate run completion from true task acceptance.
AI agentsAcceptance TestingAgent Evaluation
0 likes · 18 min read

The article details a rigorous agent evaluation framework for after-sales operations using GRPO, revealing why a 93% overall success rate masks critical failures in specific scenarios and safety guardrails, and how to structure evaluations to withstand interviewer scrutiny.
Agent EvaluationBucket AnalysisGRPO
0 likes · 18 min read

NTU researchers evaluated four configurable harnesses across five models and three benchmarks, finding harness performance depends on specific model-harness-task combinations rather than fixed rankings, with native harnesses not always optimal and cost not correlating with performance.
AI agentsAgent EvaluationAgent Harness
0 likes · 19 min read

This article presents a 10-level framework for building production-ready AI agents, centered on six critical questions about action control, capability access, conflict resolution, failure handling, knowledge reuse, and causal reasoning, plus four engineering principles emphasizing code-enforced permissions, evaluator-driven loops, identification before optimization, and falsifiable innovation.
AI agentsAgent EvaluationProduction Readiness
0 likes · 5 min read

This article analyzes how 10 leading financial institutions implement AI agents in low-tolerance scenarios, detailing their approaches to data ontology, semantic layers, multi-agent architectures, risk control, and evaluation frameworks, achieving metrics like 98.5% automated review rates and 0.003% fraud rates while ensuring auditability and regulatory compliance.
AI agentsAgent EvaluationApache Ossie
0 likes · 43 min read

HKUST researchers introduce LexAgentHallu, a hierarchical benchmark that evaluates hallucinations in legal AI agents by tracing entire reasoning trajectories, revealing that even top-performing systems exhibit hallucinations in 89% of execution traces and 68% of correct answers contain flawed reasoning.
Agent EvaluationBenchmarkHKUST
0 likes · 14 min read

Zhejiang University and Ant Group release LongDS v1.1, a benchmark for long-horizon multi-turn data analysis agents built from real Kaggle workflows; the Lite subset of 24 tasks and 777 turns shows GPT-6 Astra leading at 78.17, with detailed error analysis revealing cascading state errors as the primary failure mode.
Agent EvaluationBenchmarkEMNLP 2026
0 likes · 24 min read

This comprehensive guide covers 10 essential AI Agent interview topics, including Agent vs LLM differences, Workflow vs Agent selection, reasoning paradigms, Function Calling, MCP, error handling, memory management, context optimization, RAG pipelines, and evaluation metrics, with code examples and architectural diagrams.
AI agentAgent EvaluationFunction Calling
0 likes · 43 min read

Palantir's AI Forward Deployed Engineer (FDE) now automates execution tasks like data integration and ontology management within Foundry, while AIP Evolve enables agents to self-optimize models and prompts via eval-driven loops; Chinese enterprises similarly adopt AI FDE to parallelize delivery workflows, but human judgment remains essential for defining context, correctness, and error boundaries.
AI-FDEAIP EvalsAIP Evolve
0 likes · 21 min read

A study from AI2 and University of Washington finds that complex Harness Evolution for AI agents fails to consistently outperform simple test-time scaling methods like parallel sampling under equal compute budgets, and improvements rarely transfer to unseen tasks, questioning whether observed gains stem from genuine self-improvement or just extra attempts.
AI agentsAgent EvaluationHarness Evolution
0 likes · 14 min read

Meituan's Turing team shares a comprehensive framework for evaluating AI agents, covering multi-layered assessment (result, process, efficiency, risk), human-machine alignment via binary rubrics, seed test sets, expert knowledge integration, and the shift toward infrastructure for long-horizon agents with task-based evaluation harnesses.
Agent EvaluationEvaluation InfrastructureHuman-Machine Alignment
0 likes · 33 min read

This article details a production-grade architecture for enterprise AI agents, combining a unified harness for execution control, federated skills for domain expertise, and a virtual file system for long-task context management, drawing on Stripe's Kai platform and Deep Agents framework to address governance, security, and scalability challenges.
AI governanceAgent EvaluationAgent Harness
0 likes · 37 min read

The Shanghai finale of the Agent Observability and Optimization Close the Loop tour featured three technical sessions on building verifiable Agent optimization loops, managing Skills as strategic assets with SkillOps, and applying LangChain's Agent Engineering lifecycle from prototype to production, plus a hands-on workshop using Qoder and PawBench to demonstrate end-to-end evaluation.
Agent EvaluationAgentLoopLangChain
0 likes · 9 min read

This article distinguishes autonomous agents that dynamically plan actions from engineered agents that embed decision-making within verifiable, auditable workflows, providing criteria for choosing each approach, a three-layer production architecture, and a checklist for determining the right level of autonomy based on risk, reversibility, and audit requirements.
AI agentsAgent Evaluationagent architecture
0 likes · 16 min read

ASI‑Bench, developed by Tsinghua and leading institutions, is a benchmark that evaluates AI’s scientific autonomy by progressively reducing method guidance across four levels, revealing that current models lose up to half their scientific score without detailed instructions, highlighting the gap to true independent research.
AI autonomyASI-BenchAgent Evaluation
0 likes · 14 min read

The article presents a quantitative offline evaluation system for AI agents in real outbound-call scenarios, combining reference‑based scoring with pairwise GSB methods, addressing regression and optimization, mitigating systematic bias through judge model selection and majority‑vote adjustments, and delivering an objective, high‑efficiency benchmark.
Agent EvaluationGSBLLM-as-a-Judge
0 likes · 16 min read

This article explains why evaluating AI agents requires more than answer correctness, outlines a four‑layer evaluation framework (result, process, efficiency, risk), compares short‑ and long‑horizon agents, and presents a practical methodology that combines objective and subjective metrics, rubric binary‑ization, case management, and infrastructure requirements for scalable, repeatable agent testing.
AI agentAgent EvaluationEvaluation Framework
0 likes · 25 min read

JetBrains evaluated 225 authentic Java development tasks across multiple repositories, comparing Codex with other AI coding agents on completion rate, latency, cost, and real user behavior, and concluded that Codex should become the default agent for Java developers.
AI codingAgent EvaluationCodex
0 likes · 17 min read

The article introduces GDPevo, the first open‑source benchmark that quantifies self‑evolution in agents by generating 120 real‑world enterprise tasks, using rule‑hybrid question creation and deterministic scoring, and shows that self‑evolving agents improve accuracy by 17‑22% while reducing token consumption.
AI benchmarkAgent EvaluationContinual Learning
0 likes · 12 min read

The article explains why many AI Agent projects collapse before production due to non‑determinism, error propagation, and creative solutions, and presents LangChain’s Deep Agent evaluation framework—integrated with LangSmith, AWS Bedrock, and Pytest—to provide a reproducible, end‑to‑end testing and monitoring process.
AWS BedrockAgent EvaluationDeep Agent
0 likes · 9 min read

The article surveys DeepSeek’s Harness initiative, presenting the Binding‑Constraint Thesis, three‑stage evolution from prompt to harness engineering, the ETCLOVG seven‑layer architecture, and concrete benchmark evidence that harness‑only improvements far outweigh model upgrades, while detailing security, observability, and governance considerations for reliable LLM agents.
AI ArchitectureAgent EvaluationAgent Harness Engineering
0 likes · 12 min read

This curated collection gathers AI Agent interview questions covering fundamentals, tokenization, skill design, RAG, MCP, memory systems, evaluation methods, and practical engineering pathways, offering a complete navigation resource for backend engineers transitioning to AI roles.
AI agentAgent EvaluationInterview questions
0 likes · 3 min read


ITPUBMay 16, 2026 · Artificial Intelligence
When over 90% of a codebase is produced by AI, the authors show how a unified "people‑align → human‑machine‑align" approach, driven by evaluation agents, transforms technical debt into incremental business work, enabling continuous refactoring, AI‑friendly standards, and a sustainable engineering environment.
AI codingAI governanceAgent Evaluation
0 likes · 21 min read

When over 90% of a codebase is produced by AI, system quality hinges on constraining AI rather than speed, and this article details how a team used an agent‑based evaluation framework, unified standards, and incremental refactoring to turn 310,000 lines of AI‑written code into a maintainable, low‑debt system.
AI codingAI governanceAgent Evaluation
0 likes · 21 min read


AntDataApr 28, 2026 · Artificial Intelligence
The article presents an end‑to‑end, eight‑phase automated evaluation pipeline for large‑model agents that replaces manual bad‑case inspection with AI‑assisted pre‑annotation, cutting analysis time from a full‑day to about 30 minutes and achieving over 90 % efficiency gain while enabling iterative knowledge‑base refinement.
AI pre‑annotationAgent EvaluationAutomated Pipeline
0 likes · 20 min read

This article details a three‑day deep dive into OpenClaw, demonstrating how a self‑iterating AI assistant for bank relationship managers can be built, validated, and refined through autonomous agent communication, scheduled tasks, and memory‑driven reflection.
AI agentsAgent EvaluationMemory Architecture
0 likes · 20 min read

CATArena introduces a tournament‑style evaluation framework where AI agents iteratively code, compete, and improve across classic board games, using three‑dimensional quantitative scores to measure strategy programming, global learning, and generalization, and reveals how different LLM‑based agents learn and adapt over multiple rounds.
AI benchmarkAgent EvaluationCATArena
0 likes · 8 min read

This article explains why evaluating AI agents is essential, outlines a multi‑dimensional metric system covering performance, safety, cost and bias, describes common evaluation frameworks such as AgentBoard, AgentBench and τ‑bench, and provides step‑by‑step instructions, example datasets and code for building a robust agent assessment pipeline.
AI agentsAgent EvaluationLLM
0 likes · 35 min read

DeepSeek V3.1, released on August 19, expands context length to 128 K tokens and updates its knowledge base to July 2024, and the author’s benchmarks show its programming and agent capabilities now rival Claude 4, with detailed prompt examples, code generation demos, and performance comparisons.
Agent EvaluationClaude 4DeepSeek
0 likes · 9 min read

This in‑depth review examines Kimi K2’s new focus on agent and coding abilities, comparing its performance on 3D HTML generation, code generation, and real‑world agent tasks against Claude 4 and Gemini 2.5, while also evaluating cost, openness, and practical usability for developers.
AI codingAgent EvaluationKimi K2
0 likes · 15 min read
