Infinitus' Agent Data Flywheel: Turning Traces into Reusable Experience with AgentLoop
Infinitus built a seven-stage data flywheel using Alibaba Cloud AgentLoop to continuously improve their Lingzhi AI agent, automating trace collection, evaluation, annotation, and experience mining while adding human review gates to turn production traces into reusable prompt optimizations, failure patterns, tool paradigms, and business SOPs.
Customer Background
Infinitus (China) Co., Ltd., a member of the Li Jiji Group, operates nearly 7,000 stores across China. Their frontline store partners handle massive product inquiries, customer follow-ups, and team management daily. To assist them, Infinitus launched the AI assistant "Lingzhi" (灵知) Agent inside their app. Lingzhi relies on three proprietary knowledge bases — health, product, and operations — and serves four typical scenarios: finding materials (e.g., "find a course about immunity"), generating sales scripts (e.g., "write a follow-up for an old customer"), coaching teams (e.g., "how to onboard a newbie in month one"), and executing business tasks (e.g., "show my performance this month").
Business Challenges After Launch
After deploying Lingzhi, the team realized the agent does not automatically get better. Three core problems emerged:
Traces stuck in observability: Full DAG traces were collected but only latency was monitored; no extraction, cleansing, or evaluation pipeline existed, so traces remained raw logs.
Bad cases depended on manual review: Without evaluation thresholds, annotation had to cover everything, costing high labor and narrow coverage; annotated conclusions lived in scattered spreadsheets and never fed back into datasets.
Optimization conclusions stayed in personal experience: Prompt-change rationales were not structured or stored, forcing new maintainers to re-investigate the same issues.
Additionally, user feedback was disconnected from execution traces — a "bad response" note could not be linked to the retrieved chunks or tools used, making it unactionable.
Solution: AgentLoop-Powered Seven-Stage Flywheel
Infinitus adopted Alibaba Cloud AgentLoop , an enterprise-grade platform for observability, evaluation, and continuous optimization of agents. They split the data loop into seven stages with two cross-platform handoffs. AgentLoop handles automated data processing and experience production; Infinitus' homegrown knowledge operations system "Lingxi" (灵枢) handles human-in-the-loop annotation and review.
Stage 1: Observability Access
Applications report runtime data via SDK and probes non-intrusively, aggregated by application, session, and request. A trace_id spans the full DAG of a single response, preserving nodes, tool calls, inputs, and outputs. Multi-app, multi-environment reporting yields raw traces.
Stage 2: Observability Data Processing
A data processing pipeline runs scheduled tasks at day/hour/minute granularities (mainly offline batch, with near-real-time support). It sequentially extracts, desensitizes, deduplicates, and standardizes fields, while discarding anomalous samples. Cleansing rules are pluggable and auditable, producing cleaned samples.
Stage 3: Evaluation
AgentLoop provides built-in evaluators and supports custom Agent Judge evaluators. For Lingzhi, Infinitus configured a custom evaluator scoring four weighted dimensions: specification compliance, completeness, faithfulness to evidence, and hallucination avoidance . Higher scores mean more standard, complete, evidence-faithful, and hallucination-free answers — not subjective "looks good" ratings. The evaluator receives the entire execution trace ( trace_id, turn_index, system prompt, triggered skills, etc.), not just a query-response pair, enabling judgment of faithfulness to actual retrieved evidence. Evaluation rules are managed as versioned configuration, not code, so rule changes need no redeployment and historical scores remain explainable. Scores, reasons, and the evaluation process are all persisted for traceability.
User Feedback Correlation: Making Feedback Traceable
Infinitus solved three linkage problems:
Feedback granularity: Frontend feedback attaches to message-level IDs, not session-level, so a single faulty response in a multi-turn conversation is not diluted.
ID-to-trace mapping: The message ID retrieves the full execution DAG, revealing exactly which chunks were retrieved and which tools were called.
Prioritized evaluation: Negative-feedback samples are evaluated first (bypassing full batch cycles). Scores below threshold enter annotation; concentrated negative scenarios are prioritized for the week's evaluation batch. High-scoring positive-feedback samples become trusted positive exemplars for experience mining.
Business Attribute Staining: Propagating message_id to Every Span
To link feedback to traces, the business message_id must exist on every span. Two technical hurdles were addressed:
Field mapping: ARMS probes map LangChain metadata fields (e.g., user_id → gen_ai.user.id, session_id → gen_ai.session.id) to OpenTelemetry GenAI semantic attributes, automatically propagating them to all child spans. Custom attributes like message_id require setting Baggage on the current context to propagate to all descendant spans.
Timing: The root span is created by the probe before the business handler runs, so context set in code misses the root. Infinitus built two paths: Path A (root span): Write message_id directly into the active server span at request entry. Path B (child spans): Write Baggage once at entry; a custom SpanProcessor in on_start makes every subsequently created span (AGENT, STEP, TOOL, LLM, etc.) inherit the value.
Post-deployment, production root spans carried message_id matching the business conversation ID exactly. otel.scope.name pointed to the probe plugin, confirming automatic runtime injection rather than manual code instrumentation. message_id became the sole join key across four systems: traces, conversations, evaluation results, and annotated samples.
Stage 4: Annotation — Only Annotate What Needs Annotation
Lingxi handles human annotation. Samples scoring below the evaluation threshold enter the annotation queue; high-scoring samples become positive exemplars directly. Annotation volume is driven by the threshold, not headcount. Annotations undergo spot-checks — a single mislabel can propagate errors into experience mining and evaluator calibration. Human conclusions serve as "ground truth" for the evaluator: borderline samples judged correctly give confidence to adjust the evaluator; misjudgments often reveal flawed evaluation dimension definitions, a more powerful lever than adding more evaluators.
Stage 5: Data Feedback — Contractual Handoff
After annotation, Lingxi pushes labeled data back to AgentLoop datasets — the first cross-platform handoff. Both sides agreed on a contract covering:
Field specification: Every returned sample must include sample content, evaluation score, human conclusion, annotator, and timestamp; missing any field breaks traceability.
Cycle alignment: Feedback cadence matches the data processing cadence (day/hour/minute); batch IDs are generated per cycle; cross-cycle stitching is forbidden.
Version correspondence: Each annotation batch maps 1:1 to a dataset version, enabling rollback to the exact annotation state and evaluation criteria of that batch.
Threshold calibration: The good/bad boundary is jointly defined and versioned; neither side maintains a separate standard.
Stage 6: Experience Mining — Deriving Reusable Assets from Good/Bad Comparisons
Experience mining operates on the contrast between evaluated bad cases (sent to annotation for correction) and good cases (returned as positive baselines). The output is not vague summaries but four structured, reusable experience types:
Prompt optimization points: Which instruction caused deviation and what correction improves accuracy — fed into prompt template versioning.
Typical failure modes: Recurring error triggers and precursor signals — encoded into evaluator rules and alert thresholds.
Tool invocation paradigms: Correct call sequences, dependency chains, and fallback paths — captured in tool examples and few-shot demonstrations.
Business SOPs: Actual business handling flows and decision criteria — stored as knowledge base entries and retrieval strategies.
Experiences must be structured, searchable, and traceable to source. AgentLoop splits mining into four steps: (1) ingest traces into a trajectory library with confirmed scope; (2) configure start time and application, launch mining, monitor progress; (3) review experience content and evidence, label correctness and applicable scenarios, decide enablement; (4) integrate via API/SDK for runtime retrieval and verify hit effectiveness in debugging. Responsibilities are split: Infinitus defines trajectory sources, cleansing criteria, and mining granularity; AgentLoop records who confirmed, which version, and whether enabled.
Stage 7: Experience Integration — A Mandatory Human Gate
The second cross-platform handoff moves approved experiences into production via Lingxi in four steps:
Pull experiences: Lingxi calls AgentLoop's experience search API, landing entries in a private pending-review queue.
Human review: Reviewers verify each experience for correctness, usability, and business risk; rejected items never enter the library or get published.
Reviewed storage: Only approved experiences are written to the Lingxi experience library, recording reviewer and timestamp.
MCP publication: Lingxi wraps the library as a standard MCP Server, exposing a uniform tool name and parameter schema for internal agent runtimes; only reviewed experiences are exposed.
Rejected experiences, along with rejection reasons, are fed back to AgentLoop to refine mining strategies and recalibrate annotation guidelines. This human gate is the control point ensuring "experiences are trustworthy."
Three Practical Insights
Build the processing pipeline first, then talk data assets. Raw traces are not assets; only cleansed, evaluated, human-concluded samples are. Without a pipeline, more collection only raises storage costs.
Draw responsibility boundaries by "who owns the business outcome." Annotation and review belong to the side that best understands business consequences, not the side that finds coding easiest. Misplaced gates become empty process without judgment.
Place the gate at the experience exit, not the data entrance. Over-strict intake blocks data flow and stalls the flywheel; gating at output guarantees every experience pushed to the agent is credible.
Conclusion
Infinitus' Lingzhi Agent continuous improvement practice demonstrates a complete data flywheel: observability access → data processing → evaluation → human annotation → data feedback → experience mining → experience integration . By turning online traces into reusable, human-vetted experiences — prompt fixes, failure patterns, tool paradigms, and business SOPs — the flywheel ensures each iteration produces a better response than the last.
Signed-in readers can open the original source through BestHub's protected redirect.
This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactand we will review it promptly.
How this landed with the community
Was this worth your time?
0 Comments
Thoughtful readers leave field notes, pushback, and hard-won operational detail here.
