Skip to content
Governance & Evaluation of Load-Testing AI Agents: A Closed-Loop Practice with Langfuse

Governance & Evaluation of Load-Testing AI Agents: A Closed-Loop Practice with Langfuse

The article details a closed-loop governance framework for load-testing AI agents, combining Langfuse traceability, fixed-evidence replay via CLI Mock, and LLM-as-a-judge evaluation to iteratively improve agent quality—demonstrated by a Dubbo debugging case that raised solution accuracy from 61.67% to 96.15% while cutting latency and cost.

DeWu Technology
DeWu Technology
DeWu Technology

Load-testing debugging is the most feedback-heavy stage: the platform supports 10 protocols and 30+ script components, and newcomers often hit errors when orchestrating scripts or configuring parameters. Traditional troubleshooting requires manually checking scenario configs, JMX scripts, request/response payloads, JMeter logs, and traces—a long, laborious chain.

To automate this, an intelligent analysis Agent was integrated into the existing flow: Debug page → Copilot → Java business gateway → Agent SDK runtime → Skills & CLI → platform data & local scripts. Rules and programs first gather evidence and generate candidate causes; the model then reviews context and outputs a solution. Langfuse records the full analysis process for later troubleshooting and evaluation.

System architecture: Copilot, Java gateway, Agent SDK, Skills/CLI, platform data, Langfuse
System architecture: Copilot, Java gateway, Agent SDK, Skills/CLI, platform data, Langfuse

Governance Challenges & Target Loop

After rolling out to real business, three governance problems emerged:

Feedback hard to collect: users report after analysis ends, missing original inputs, scene configs, and execution records.

Evaluation not unified: different scenes, versions, and user focuses lack a common standard.

Process not transparent: without seeing tool calls, evidence sources, and reasoning, it's impossible to pinpoint whether the fault lies in evidence collection, rule handling, or model analysis.

The target continuous governance loop: User feedback → associate Trace → reproduce & locate issue → fix Skill/rule/evidence logic → solidify regression case → evaluate & release . Every confirmed issue must leave reusable troubleshooting experience and repeatable regression samples.

Langfuse Foundation: From Process Recording to Troubleshooting

Langfuse does not participate in diagnosis; it retains Agent inputs/outputs, tool calls, latency, and model cost so a single analysis can be replayed, inspected, and compared. When a user reports an unsolved problem, the corresponding Trace lets engineers return to the exact analysis scene and identify the deviating step.

Langfuse architecture: SDK, Web/API, Worker, storage, Trace query, dataset management, scoring analysis
Langfuse architecture: SDK, Web/API, Worker, storage, Trace query, dataset management, scoring analysis

Feedback Example: From User Report to Governance Queue

A report "Agent solution didn't solve the problem" includes scene info, original request, solution gaps, and a langfuseId (Trace ID). The troubleshooting steps:

Locate call: use Trace ID to find the record; verify timestamp, business scene, user input, Agent version.

Reproduce chain: view model and tool call order; compare user request, analysis process, final solution to confirm actual executed steps.

Verify evidence: expand key tool nodes; check call parameters, execution results, exceptions to judge whether the issue stems from missing evidence, tool failure, rule deviation, or model misuse of evidence.

Register governance case: record solution gap, preliminary cause, and items to re-verify; add to Annotation Queue as source for later fixes and regression samples.

Feedback UI showing scene info and langfuseId
Feedback UI showing scene info and langfuseId
Trace view: model and tool call sequence
Trace view: model and tool call sequence

Evaluation System: Verifying Agent Quality with Fixed Evidence

After a fix, the change must be validated on the same problem scene. Core principle: different Agent versions face identical problems, business context, and diagnostic evidence; only the factor under test changes. Langfuse manages evaluation datasets, experiments, and scores; CLI Mock replays fixed business evidence during back-testing.

Fixed Scene & Back-Testing

The platform solidifies user question, scene config, JMX script, logs, request/response into an evaluation Case, each with a human-verified reference answer. During back-test, CLI Mock returns evidence consistent with the original problem, preventing live data drift from affecting version comparisons. The real Agent still runs: it calls tools, reads Mock evidence, analyzes, and produces a solution; the evaluator then scores the actual output.

When verifying tuning effects, fix dataset, Mock evidence, model, evaluator, and execution budget; only modify the target Skill, rule, or analysis flow. When comparing candidate models, fix Agent implementation and evaluation config; only swap the model. For high-variance Cases, repeat runs and observe multi-round results—never judge by a single peak score.

Evaluation flow: Fixed problem & evidence → Agent re-analyzes → Auto-scoring → Result comparison & regression.

Evaluation flow diagram
Evaluation flow diagram

Metrics: Solution Quality, Latency, Cost

Results are observed from three angles: quality (primary), latency, and cost. When quality is close, trade-offs consider latency and cost. Formal comparison looks beyond averages: P95 latency, cost variance, and multi-run stability to avoid averages masking outliers.

Metrics dashboard: quality, latency, cost
Metrics dashboard: quality, latency, cost

Evaluator: LLM-as-a-Judge with 5-Tier Scoring

The evaluator uses a model to judge semantic consistency and key-content coverage against a human-curated expected answer. It receives three inputs: input (user question for context), expectOutput (human reference answer, primary scoring basis), actualOutput (Agent-generated solution). The reference answer is not a unique gold standard; the evaluator does not require reproducing reasoning, wording, structure, evidence order, or style.

The evaluator returns only a legal score (0, 0.25, 0.5, 0.75, 1) with no rationale. Human reference answers set the unified standard and evaluation boundary; high/low/disputed scores are sampled for review, missing valid solution paths are added, then the unified version re-evaluates. Online fix effect is ultimately confirmed by user re-debugging.

The full evaluator prompt (abridged) defines scoring rules: primary scoring points from expected output; full solution with major points covered → high score even if minor points missing; different expression but semantic consistency → no penalty; missing key evidence affecting credibility → score ≤ 0.75; contradictory content → penalty; explicit user format/field/action requirements not met → penalty; never use own knowledge to override expected output core conclusions.

You are a strict, objective answer correctness evaluator.
Your task is to assess whether the Agent's actual output semantically fulfills the user task based on the human-corrected reference answer.
Reference answer is not the only standard answer, nor does it require the actual output to replicate reasoning, wording, structure, evidence order, or style.
Variables:
- <user_input>: original user question for understanding goal, context, constraints.
- <expected_output>: human-corrected reference answer, primary scoring basis.
- <actual_output>: Agent-generated answer, evaluation object.
Internal analysis steps:
1. Identify user's real problem, goal, constraints from <user_input>.
2. Extract scoring points from <expected_output>: major (core conclusions, necessary constraints, key facts/evidence) and minor (auxiliary info, supplementary explanations, non-essential details).
3. Judge if <actual_output> semantically covers these points.
4. Judge if <actual_output> solves the user's main problem.
5. Identify missing major/minor points.
6. Check for contradictions with reference core conclusions, key facts, or explicit user constraints.
7. Select score per criteria.
Scoring rules:
- Base on major points in <expected_output>.
- If actual output fully solves main problem and covers major points, even with missing minor points → high score.
- Different expression/order/reasoning but semantic consistency → no deduction.
- No requirement to replicate reference reasoning chain, evidence order, or style.
- Missing minor points → no major deduction.
- Missing key evidence directly affecting conclusion credibility/verifiability/executability → score ≤ 0.75.
- Extra content not in reference but not contradictory/misleading → no deduction.
- Contradictions with core conclusions/key facts/user constraints → deduction.
- Requirements only in <expected_output> without explicit necessity tag → reference only, not hard deduction.
- Explicit user format/field/action requirements not met → deduction.
- Never replace or overturn <expected_output> core conclusions with own knowledge.
- <user_input> only for understanding task goal/constraints, not independent standard.
Score standards:
1: fully solves main problem, complete major points, semantically correct, no substantive errors/omissions/contradictions; minor points may be missing.
0.75: solves main problem, core conclusion correct, covers most major points; only minor omissions, incomplete expression, small issues not affecting main conclusion.
0.5: covers some major points, helps solve main problem, but important content missing, incomplete, local factual errors, local contradictions, or fails important explicit user constraints.
0.25: covers few major points, core conclusion incomplete/unclear, obvious errors, limited help.
0: opposite to reference core conclusion, irrelevant, empty, erroneous refusal, or covers no valid expected content.
Constraints:
- If core conclusion opposite to reference → score must be 0.
- If main problem not solved → score ≤ 0.25.
- If solves main problem but misses important content/steps/explicit constraints → score ≤ 0.5.
- If fully solves main problem, only minor omissions/small issues → score 0.75 or 1.
- Only if fully solves main problem, complete major points, no substantive errors/contradictions → score 1.
- Score only from {0, 0.25, 0.5, 0.75, 1}.
Evaluation data follows in <evaluation_data> with user_input, expected_output, actual_output.
Return only one of the five numeric values. No analysis, reasons, text, tags, JSON, Markdown, or code blocks.

Adding dataset items: in the Queue, locate items needing evaluation, edit required input and expected output , then add to the evaluation set.

Adding dataset items in Langfuse Queue
Adding dataset items in Langfuse Queue

Tuning Case: Solidifying Dubbo Troubleshooting Experience into Agent

Dubbo debugging depends on environment, service/method config, scene orchestration, and execution state. The team distilled manual troubleshooting experience into Agent component knowledge, validation rules, and analysis flow , making solutions land on concrete config items, and verified via fixed-case back-testing.

Dubbo Config Checklist

Dubbo scenario config checklist: environment, registry, service, method, orchestration, execution state
Dubbo scenario config checklist: environment, registry, service, method, orchestration, execution state

Three Exception Examples: From Error to Config Check

Consumer is shutting down → verify addressing & exit cause. Log marks dubbo://XXX:8848 as provider; check if Nacos was mistaken as direct address, and if registry protocol and namespace are correct. Combine with JMeter logs to verify task stop reason; cannot conclude root cause from "shutting down" alone.

No provider available → verify service discovery conditions. Example query xxx/SpuXXXXService:1.0; check environment, namespace, group/version against actual registration info; distinguish config mismatch, instance unavailable, or routing filter. Refer to official troubleshooting guide.

Failfast / $invoke → verify method contract. Request calls queryV6; provider's methods list includes V5 but not V6. Verify target environment's method, parameter types, service version; modify config after confirmation; cannot simply replace V6 with V5. Refer to generic invocation guide.

Agent-Side Improvements: Knowledge, Rules, Analysis Chain

Agent architecture: CLI collects, Python does deterministic processing, LLM does reasoning
Agent architecture: CLI collects, Python does deterministic processing, LLM does reasoning

CLI collects, Python handles deterministic processing, LLM handles reasoning. Python parses JMX, calls CLI to fetch contracts, executes validations, generates problem candidates, evidence indices, and candidate reports. LLM reads component knowledge and analysis guidance, combines with evidence to confirm or correct candidates, outputs root cause and fix plan. Python rule hits ≠ final diagnosis.

Back-Test Results: Dubbo Specialty Drives Overall Improvement

Dubbo specialty evaluation results
Dubbo specialty evaluation results

Per the new version results, Dubbo specialty evaluation rose from ~ 50% to 90% (~40 percentage points), a major contributor to overall gains. Specialty, overall phase intervals, and the single-round 96.15% in section 5.1 correspond to different statistical scopes; each is labeled with dataset and version when presented.

Before/after solution comparison (images show concrete diff): optimized solution includes precise config checks and actionable modifications.

Pre-optimization solution
Pre-optimization solution
Post-optimization solution
Post-optimization solution

Model Comparison: Comparing Candidates Under Identical Conditions

Agent knowledge, rules, and execution flow all affect results, so the model itself needs isolated comparison. Three candidates: deepseek-v4-flash, qwen3.8-flash, glm5.3-flash . The table below compiles historical runs from the experiment "HTTP & Dubbo scenarios, round 4" for stage-wise observation. Dataset, Agent, and evaluator versions still need cross-checking; results are provisional references.

Model comparison table: DeepSeek highest score, GLM lowest latency, Qwen/GLM cost 0.00 (unverified)
Model comparison table: DeepSeek highest score, GLM lowest latency, Qwen/GLM cost 0.00 (unverified)

From this data: DeepSeek scores highest; GLM has lowest average latency. Qwen and GLM show cost 0.00, but it's unclear whether cost is truly zero or billing data not yet integrated—thus cost comparison is invalid. These records only reflect stage-wise performance under then-current versions, insufficient for formal model selection. Formal comparison requires same Agent version, same dataset, same Mock evidence, same evaluator, repeated runs per model. Selection priority: solution quality; if close, then latency, cost, multi-run stability.

Raw evaluation run records for verification
Raw evaluation run records for verification

Key Takeaways: Making Troubleshooting Experience Verifiable Agent Capability

Ground subjective feedback in concrete evidence with Traces. From user request, tool I/O, to final solution—locate the exact step where omission, misinterpretation, or execution failure occurred to determine improvement target.

Assign codable experience to Python, semantic judgment to LLM. Verifiable constraints (environment addressing, service/method contracts) become rules; LLM cross-checks candidates against business intent and raw evidence to produce evidenced solutions.

Retain fixed problems, continuously validate subsequent versions. Mock freezes the scene; human expectations unify standards; evaluation watches quality, latency, cost together. Real-world user debug results close the loop, turning each troubleshooting episode into reusable, verifiable capability for the next iteration.

Original Source

Signed-in readers can open the original source through BestHub's protected redirect.

Sign in to view source
Republication Notice

This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactadmin@besthub.devand we will review it promptly.

Dubboload testingAI AgentEvaluationGovernanceLangfuseLLM-as-a-JudgeCLI Mock
Reader feedback

How this landed with the community

Sign in to like

Rate this article

Was this worth your time?

Sign in to rate
Discussion

0 Comments

Thoughtful readers leave field notes, pushback, and hard-won operational detail here.