Close the Loop Finale: Agent Evaluation, SkillOps & LangChain Engineering Practices

The "Agent Observability and Optimization · Close the Loop" three-city tour concluded at its Shanghai station on September 4, co-hosted with LangChain and attended by nearly 100 developers. The event centered on the theme "Optimize / 优化与进化" and featured three technical sessions plus a hands-on workshop.

Session 1: End-to-End Agent Optimization Loop

Presented by Liu Hang (Wang Xi), R&D Engineer at Alibaba Cloud AgentLoop. The talk introduced a verifiable optimization loop composed of seven stages:

Unified trace ingestion with consistent schema.

Observability that drills from global trends down to raw evidence of anomalous steps.

Dataset freezing to make evaluation reproducible.

Data processing that assembles raw call chains into evaluable trajectories.

Evaluation that not only scores but also identifies where errors occur, why, and who should fix them.

Experimentation using same-sample A/B testing to ensure a single variable.

Optimization that selects the smallest lever on either the Agent side or the platform side.

Four metric categories — quality, efficiency, cost, and safety — run through the entire loop. Runtime results and user feedback flow back into the sample pool, forming a continuous evolution cycle. Experience (Experience) is also managed uniformly: from fact parsing and pattern mining to quality gating, matching recall, and effect write-back, ensuring every piece of experience has a source, boundaries, and an exit mechanism.

Session 2: Skill Quality Improvement and Asset Precipitation — SKU Operations in the AI Era

Presented by Zhong Yu, R&D Engineer at Lingyue Technology. The session shared the SkillOps practice: treating Skills as discoverable, measurable, and iterable strategic assets. A "four-square" AI infrastructure — Model Square, MCP Square, Skill Square, and Plugin Square — enables Skills to flow across teams. The Skill lifecycle is broken into six phases:

Discovery: prioritize reuse of existing Skills.

Creation: one-click generation of prompts, configurations, tests, and documentation.

Release: automatic listing via Git repository.

Data collection and processing: using AgentLoop's Loongsuite collector to capture invocation data and attribute it to Skills without user awareness.

Evaluation: offline experiments on real datasets comparing with_skill vs. without_skill, quantifying functional correctness, capability gain, and token cost.

Iteration: targeting failure scenarios with pre-labeling and human review to produce high-confidence datasets for the next version.

This approach gives users zero-barrier reuse, developers real-time data dashboards, and managers an enterprise-grade core AI asset library from previously scattered tools.

Session 3: LangChain Agent Engineering — ADLC Practice from Prototype to Production

Presented by Zhang Haili, LangChain Ambassador. The talk framed Agents as a new software engineering challenge: non-deterministic outputs, invisible behavior, traditional tests unable to measure quality, and insufficient infrastructure for long-running tasks. Agent Engineering is the discipline of iterating non-deterministic systems into reliable experiences, with a core lifecycle of Build → Observe → Evaluate → Deploy.

LangSmith serves as the Agent Engineering Platform, connecting downward to the LangGraph runtime, LangChain framework, and Deep Agents harness, and upward providing observability, evaluation, deployment, and no-code building capabilities.

Unlike traditional APM, Agent observability uses Trace as the source of truth, capturing prompt-response pairs, multi-turn context, tool calls, and reasoning steps. An Insights Agent automatically discovers usage and error patterns, while online evaluation continuously monitors quality, safety, and format compliance. Evaluation supports offline dataset regression, LLM-as-judge, human annotation queues, and prompt A/B testing in the Playground. Deployment offers one-click publishing, Agent Registry, Assistants API, and LangSmith Studio. The entire stack remains model-, tool-, and database-neutral, avoiding vendor lock-in and allowing enterprises to swap infrastructure flexibly.

Hands-On Workshop: Closing the Qoder + PawBench Evaluation Loop with AgentLoop

Led by Huang Xiaolong (Yi Xingchen), R&D Engineer at AgentLoop. Participants used Qoder and PawBench to experience the full cycle: "Coding Agent execution → Trace collection → Dataset processing → Agent evaluation." The workshop fixed two PawBench tasks (T036 and T063) as samples, first running a baseline then injecting the fact_extraction_exactness experience. Loongsuite Pilot reported Qoder traces to AgentLoop. An Agent evaluator based on PawBench ground-truth accuracy was created, and a pipeline combined GT, input, output, and the full agent_trajectory into an evaluation dataset. Batch evaluation on the AgentLoop product page then compared the effect before and after experience injection.

Tour Retrospective

From the Beijing station on August 21 ("Observe / 观测") to the Shenzhen station on August 29 ("Evaluate / 评估") and the Shanghai station on September 4 ("Optimize / 优化"), the three-city tour delivered a unified message: an Agent is not a one-time release but a production system requiring continuous observation, evaluation, and optimization. Observation makes problems visible, evaluation grounds improvements in evidence, and optimization ensures every change withstands real-world validation.