6 Agent Baseline Metrics & 4 RAG Metrics: Definitions, Formulas, and Principles Explained

Why Agents Need a Separate Baseline Metric Suite

Traditional LLM evaluation metrics (relevance, faithfulness, hallucination rate) assess linear pipelines where the final output largely reflects system quality. Agents, however, operate in dynamic loops following a "think → act → finish" logic. The same user request can produce different execution paths across runs. Failures may arise in planning, tool selection, parameter generation, or execution efficiency — often leaving no trace in the final text. Therefore, Agents require metrics that score the entire execution trajectory, not just the final answer.

The six metrics map to three Agent runtime layers:

Reasoning layer (planning): PlanQuality, PlanAdherence

Tool layer (action): ToolCorrectness, ArgumentCorrectness

Execution layer (closure): TaskCompletion, StepEfficiency

Each layer gets two metrics, each targeting a distinct failure mode — six metrics covering six failure types.

Framework Foundation: Two Evaluation Scopes × Three Runtime Layers

Trajectory-level metrics

Evaluate the entire ordered trace as a whole. Metrics: PlanQuality, PlanAdherence, TaskCompletion, StepEfficiency. They consume the full plan, every LLM decision, every tool call and result.

Component-level metrics

Evaluate only the single LLM span that produced a decision. Metrics: ToolCorrectness, ArgumentCorrectness. Mounted via @observe(type="llm", metrics=[...]) on the specific decision point.

# Trajectory vs component metric mounting example
from deepeval.metrics import (
    PlanQualityMetric, ToolCorrectnessMetric
)
# 1. Trajectory-level: pass via evals_iterator to assess whole trace
trajectory_metrics = [PlanQualityMetric()]
# 2. Component-level: decorate the specific LLM call function
@observe(type="llm", metrics=[ToolCorrectnessMetric()])
def call_llm_for_tool_selection(context):
    # only evaluates this tool-selection step
    return llm.invoke(context)

An Agent can mount both types simultaneously without conflict. Trajectory metrics go into evals_iterator; component metrics attach via decorators.

Six Baseline Metrics Broken Down

3.1 Reasoning Layer: PlanQuality & PlanAdherence

PlanQuality (Plan Quality)

Assesses the plan's intrinsic rationality. Extracts user task and Agent-generated plan from the trace, feeds both to an LLM judge, outputs an alignment score: AlignmentScore(Task, Plan). Dimensions: logical self-consistency, reasonable task decomposition, correct dependency ordering.

Engineering boundary: If no explicit plan is extracted from the trace, the metric defaults to 1.0. It only evaluates emitted plans, not implicit reasoning. Agents that never surface a plan text render this metric ineffective.

PlanAdherence (Plan Adherence)

Measures whether actual execution steps match the prior plan. Formula: AlignmentScore((Task, Plan), Execution Steps). The judge receives user task, Agent plan, and executed steps, comparing action sequence against planned steps. Official guidance recommends pairing both metrics: a high-quality plan that drifts during execution, and a flawed plan executed faithfully, both represent business risks.

Academic lineage: Agent GPA paper's Plan Quality and Plan Adherence. Also appear in RigorBench as Planning Fidelity and Plan-Execution Alignment. Both are referenceless trajectory metrics; evaluation ceiling depends on trace plan completeness.

3.2 Tool Layer: ToolCorrectness & ArgumentCorrectness

ToolCorrectness (Tool Correctness)

Judges whether tool selection and invocation are compliant. Logic: correctly_used_tools / total_tool_calls. Compares actual tool calls in the LLM span against the dataset's expected_tools. Offers five strictness levels (loose to strict): match tool name only → validate input parameters → validate return results → enforce call order → require exact tool-call list match.

Default deterministic comparison (no LLM judge), low cost, high reproducibility. Hidden behavior: when available_tools is supplied, an extra LLM judge checks if the chosen tool is globally optimal; final score is the minimum of deterministic score and judge score.

ArgumentCorrectness (Argument Correctness)

Validates whether tool input parameters are sensible. Formula: correctly_generated_input_params / total_tool_calls. Fully LLM-driven, referenceless. Does not compare against fixed expected values; instead judges if parameters can be reasonably derived from preceding context. Fits scenarios where parameter values cannot be enumerated but must align semantically with user intent (e.g., deriving date, origin, destination from "book a flight from New York to Paris on March 15"). Academic counterpart: Parameter Appropriateness in RecToolBench.

Both metrics mount on the LLM decision span at component granularity, precisely catching a classic failure: right tool, hallucinated parameters.

# ToolCorrectness configuration: strictness levels
from deepeval.metrics import ToolCorrectnessMetric
# Strict mode: tool name + arguments must match
metric = ToolCorrectnessMetric(
    strict_mode=True,
    evaluation_steps=["Check if arguments match expected values"]
)
# Loose mode: only tool name needs to match
metric_loose = ToolCorrectnessMetric(strict_mode=False)

3.3 Execution Layer: TaskCompletion & StepEfficiency

TaskCompletion (Task Completion)

Top-level metric for the whole Agent evaluation suite. Formula: AlignmentScore(Task, Outcome). Answers whether the user's request was satisfied. Task info can be inferred from trace or passed explicitly. Score 1.0 = fully achieved. Highest consensus across Agent benchmarks (T-Eval, ToolBench); corresponds to Goal Fulfillment in GPA paper. It is a trajectory metric, not a black-box metric: scoring incorporates all steps and intermediate results, not just final text.

StepEfficiency (Step Efficiency)

Detects superfluous actions in the execution chain. Formula: AlignmentScore(Task, Execution Steps). Penalizes redundant tool calls, unnecessary reasoning loops, any non-essential steps. Corresponds to Execution Efficiency in GPA paper.

Practical diagnostic combo: high TaskCompletion but low StepEfficiency → Agent succeeds but with optimizable path; optimization direction: trim steps while preserving success rate. Used in cost-sensitive production. Note: academic efficiency evaluations often add token cost and latency; DeepEval's default only judges semantic necessity — business resource costs need separate instrumentation.

Systematic Comparison: Six Agent Metrics vs. Four RAG Metrics

4.1 RAG Baseline Metrics (Ragas / EACL 2024)

Faithfulness: Answer stays grounded in retrieved context, no unsupported fabrication.

Answer Relevancy: Final answer matches user query.

Context Precision: Ranking quality of relevant chunks in retrieval results.

Context Recall: Whether all required information was retrieved.

Evaluation evidence: static triple of user query, retrieved context, final answer. DeepEval provides same-lineage metrics (ContextualRelevancyMetric, FaithfulnessMetric, etc.).

4.2 Four Key Differences

Evaluation object differs. RAG metrics assess static retrieval-generation pipeline alignment (unidirectional, no branching, no external actions). Agent metrics assess dynamic execution traces (trace span trees) containing plans, decisions, tool inputs/outputs, retries. RAG evaluates output quality; Agent evaluates path quality.

Captured failure modes differ. RAG focuses on retrieval/generation faults: irrelevant chunks, missing context, hallucination ungrounded in context, query-answer mismatch. Agent metrics capture planning/execution-loop faults: flawed plan decomposition, execution deviating from plan, wrong tool selection, parameter hallucination, redundant calls, futile retry loops. Example: same Agent, two runs produce identical final text — one direct, one with wasteful retries. RAG faithfulness/relevancy both score high; StepEfficiency and ArgumentCorrectness expose the hidden pathology.

Layering granularity differs. RAG metrics split into retrieval-layer + generation-layer; no dedicated planning layer. Agent metrics use three layers: reasoning/planning + action/component + global execution. Layering purpose: fault localization — metric mounts at the exact span/tool call where failure occurs. RAG metrics struggle to pinpoint whether a low score originates in retrieval or generation.

Judge alignment targets differ. RAG's LLM judge aligns to a single ground truth: retrieved context facts. Agent metrics have multiple alignment sources: user goal (TaskCompletion), Agent's own plan (PlanQuality, PlanAdherence), tool schema & business constraints (ToolCorrectness, ArgumentCorrectness).

4.3 Combining Both Metric Suites

They are complementary, not mutually exclusive. Many Agents embed RAG retrieval; best practice: collect and alert on both suites separately.

Ragas metrics: evaluate factual quality of each retrieval-generation round (faithfulness, relevancy, context precision/recall).

DeepEval's six Agent metrics: evaluate planning logic, tool-calling behavior, trajectory efficiency.

In short: RAG metrics govern "how well it speaks"; Agent metrics govern "how correctly and efficiently it acts". Mixing them in one dashboard muddies root-cause analysis.

# Hybrid system evaluation: RAG + Agent
from deepeval.metrics import FaithfulnessMetric, TaskCompletionMetric
# 1. RAG metrics: assess retrieval content quality
rag_metrics = [
    FaithfulnessMetric(),
    ContextualRelevancyMetric()
]
# 2. Agent metrics: assess task execution flow
agent_metrics = [
    TaskCompletionMetric(),
    StepEfficiencyMetric()
]
# 3. Combined evaluation: view reports separately
test_case.evaluate(rag_metrics + agent_metrics)

Closing: Metric Selection & Academic Backing

For engineering decisions — CI gates, regression testing — this suite adapts well: native pytest compatibility, trace-bound, strong fault localization.

For continuous production monitoring: TaskCompletion and ToolCorrectness suit core KPI dashboards; Plan-series metrics fit offline regression and version comparisons.

Summary: These six metrics are the engineering distillation of established academic ideas (Agent GPA, RigorBench, RecToolBench), carrying paper-backed foundations and growing industrial consensus. They represent a mainstream baseline for Agent trajectory evaluation, ready for production adoption.

Typical hybrid RAG+Agent systems should layer evaluation pipelines as described, letting each suite own its domain.