Langfuse Three-Level Agent Evaluation: Outcome, Trajectory & Step Framework Principles & Implementation

1. Decoupling Monitoring (Observability) from Evaluation

Langfuse separates Monitoring (fact collection) from Evaluation (quality judgment). Monitoring captures the full execution trace as a tree of Observations: user input, each LLM generation, tool calls with parameters, retrieval contexts, token usage, latency, and error stacks. Evaluation then runs independent checks on this trace.

Example: For a flight-booking request "Book a flight from Beijing to Shanghai tomorrow", Monitoring records nodes for intent recognition, slot extraction, flight search, and answer generation. Evaluation produces four independent judgments:

Intent matches book_flight Slot extraction accuracy (date, origin, destination)

Tool call compliance and parameter validity

Output satisfaction of business goal

Only checking the final output masks internal defects—an agent may take a convoluted, error-prone path yet accidentally produce the right answer, leading to "inflated metrics" that hide instability.

2. Langfuse's Four Core Components

2.1 Trace: Full-Chain Trajectory Storage

A Trace represents one complete user request lifecycle, composed of nested Observations (spans). Each Observation captures:

Structure: call order and nesting, reconstructing the agent's reasoning path

Inputs/outputs: prompts, model responses, retrieved documents, tool arguments and results

Runtime metrics: latency, token consumption, cost, error stacks

Business tags: user ID, session ID, custom metadata

Mounted scores: Evaluator-produced scores attached to the whole Trace or any individual Observation

2.2 Dataset: Standardized Test Sample Library

Dataset samples are fixed baselines for regression. Each sample contains input, expectedOutput (intent, slots), and metadata. Example:

{
  "input": "帮我订明天北京到上海的机票",
  "expectedOutput": {
    "intent": "book_flight",
    "slots": {"date": "明天", "from": "北京", "to": "上海"}
  },
  "metadata": {"task": "book_flight", "source": "manual"}
}

Datasets evolve continuously: initial samples from production traffic sampling + handcrafted golden cases; ongoing addition of low-score traces, judge-uncertain cases, and production bad-cases after human review; weekly augmentation and deduplication to match real user distribution.

2.3 Evaluator: Scoring Execution Engine

Evaluators consume Traces/Observations and expected outputs, emit structured Scores. Three implementation flavors run in priority order:

Code Rules (preferred) : Deterministic functions for objective checks—JSON/Schema validation, required fields, tool whitelists, parameter ranges, business hard constraints (e.g., date not in past). Zero token cost, millisecond latency, 100% reproducible.

LLM-as-a-Judge (semantic fallback) : Independent LLM for subjective judgments—relevance, faithfulness, tone compliance. Production requirements: fixed evaluation boundary, explicit rubric, enumerated labels, forced JSON output; must achieve ≥0.85 agreement with human labels on 30-50 calibrated samples before gating releases.

Human Annotation (high-risk boundary) : For rules that can't be coded, low-confidence judge outputs, or compliance-critical scenarios (refunds, rebooking). Human labels serve both as immediate ground-truth scores and as golden/bad-case feedstock for Dataset.

Scores attach at four granularities: whole Trace, single Observation (recommended), Session, or Experiment Run. Four data types: NUMERIC (0-1), BOOLEAN, CATEGORICAL (e.g., Relevant/Partial/Irrelevant), TEXT (human rationale). Mounting on Observations enables pinpoint root-cause attribution.

2.4 Experiment: Versioned Regression Snapshots

An Experiment bundles a fixed Dataset + task function + Evaluator set, runs batch evaluation, and produces a snapshot with:

Per-sample details: input, agent output, every score, failure reason

Aggregated metrics: pass rate, mean F1, faithfulness mean, etc.

Version diff vs. baseline: metric deltas, regressions

Cost/performance stats: avg tokens, latency, failure ratio

CI pipelines invoke Experiments as release gates: any golden-sample failure, metric regression beyond threshold, or cost/latency spike blocks the PR. Each stable release becomes the new baseline.

3. Three-Level Evaluation Framework

3.1 Outcome (Result Layer)

Validates final user-facing output: task success, answer correctness. Business red-line metric for release gating. Weakness: coarse attribution—failure only tells "request failed", not whether intent, retrieval, tool, or generation was at fault.

3.2 Trajectory (Path Layer)

Validates the decision sequence: tool call order, redundant calls, exception handling, retry logic, replanning grounded in observed results. Metrics: tool_accuracy, step_efficiency, retry_rate. Catches "right answer, wrong path" (e.g., querying hotels then weather before flights). Weakness: a perfect path can still yield a wrong final answer.

3.3 Step / Observation (Node Layer)

Scores each atomic span independently: retrieval relevance ( context_relevance), generation faithfulness ( faithfulness), JSON validity ( json_valid), parameter schema compliance. Enables surgical root-cause localization. Weakness: all steps passing doesn't guarantee end-to-end task completion due to inter-step coordination gaps.

3.4 Joint Verification Logic

All three layers must be collected simultaneously and cross-constrained:

Outcome alone → random lucky paths inflate scores

Trajectory alone → business goal unverified

Step alone → coordination failures missed

Combined gate: Outcome ensures delivery, Trajectory governs decision quality, Step enables targeted optimization.

4. Evaluator Implementation Paradigms

4.1 Code Evaluator First

Deterministic functions for five scenario classes: JSON/Schema legality, required fields, tool whitelists, parameter types/ranges, business hard rules. Advantages: zero LLM cost, ms latency, deterministic, easy debugging. Boundary: cannot assess semantics, intent alignment, hallucinations.

4.2 LLM-as-a-Judge for Semantics

Production Judge requires four fixed configs: evaluation boundary, judgment rubric, enumerated labels with definitions, forced JSON output. Calibration on 30-50 human-labeled samples; <0.85 agreement → trend observation only; ≥0.85 → release gate. Judge's own trace is logged for auditability. Best practice: use a different model vendor than the agent to avoid correlated bias.

4.3 Human Annotation Flywheel

Two queues: uncodable rules, low-confidence/high-risk judge outputs. Human labels provide immediate ground truth and feed back into Dataset as golden/bad cases, creating a self-improving loop: production traces → auto-scoring → low/uncertain samples → human review → Dataset enrichment → next Experiment regression covers new cases.

5. Integration Architecture

5.1 Minimal Instrumentation

Business code only emits Traces via SDK; evaluation runs asynchronously off the critical path. Standardized span names: understand, retrieve, tool, answer —enables rule matching.

5.2 Mounting Rules

Configuration maps target Observation name → Evaluator(s) → trigger condition. Example: generate-answer spans run faithfulness + answer_relevancy; search_flight tool spans run parameter validation.

5.3 Async Sampling

Scoring moves to a consumer queue; main request only persists Trace. Initial 5-10% sampling, adjustable after stability.

6. Offline/Online Closed Loop

6.1 Offline: Pre-Release Verification

CI runs Experiment on golden Dataset, compares metrics against stored baseline snapshot. Failures, regressions, cost/latency anomalies break the pipeline.

6.2 Online: Post-Release Monitoring

Continuous Trace collection; code Evaluators run full-volume on hard rules; Judge samples semantic checks; user feedback flows back; low-score traces enter annotation queue. Recurring anomaly patterns (ambiguous references, tool timeouts, injection attacks, multi-turn state loss) are human-confirmed and added to Dataset for next regression cycle.

7. Small-Team Unified Langfuse Architecture

All capabilities—Trace storage, Dataset hosting, Experiment scheduling, built-in code Evaluators, Judge templates, annotation queues, dashboards, version reports—reside in Langfuse. Teams implement a thin Evaluator abstraction layer ( evaluate(span, trace) -> list[Score]) with three concrete implementations: rule, judge, business metric. Same evaluator code runs in local debug, CI Experiment runner, and online consumer worker. Benefits: logic lives in repo (UI only configures), local step-through debugging, zero external dependencies for new metrics.

8. Four-Phase Rollout for Small Teams

Observability Foundation : Full Trace instrumentation with fixed span names; one week of manual trace review to catalog defect types.

Dataset + Hard Rules : 50-80 production samples + 20 golden cases; deploy code Evaluators for JSON, tool calls, slots, business rules.

Judge + CI Gate : Custom Judge for semantics; PR merge triggers golden-sample regression via Experiment with pass/fail gate.

Online Flywheel : Enable sampled scoring; low-score → annotation → Dataset refresh; monthly baseline recalibration.

Order is fixed: observability before Judge, otherwise scores lack attributable nodes.

9. Summary

Agent evaluation quantifies behavioral quality. Langfuse closes the loop: Trace reconstructs execution, code rules catch hard defects, Judge handles semantics, human annotation calibrates edge cases, bad cases enrich Dataset, CI blocks regressions. This graduates agent projects from prototype to engineered delivery.