From Manual API Calls to Thinking Agents: DeWu Recommendation System Diagnosis

Background and Motivation

DeWu’s recommendation system historically relied on manual log inspection and expert‑driven debugging, which caused low efficiency, high MTTR, and knowledge loss when personnel changed. The team identified three recurring limitations of traditional diagnostic approaches: (1) linear, script‑based SOPs that cannot keep pace with rapid product changes, (2) pure LLM agents with high inference cost and latency, and (3) isolated expert knowledge that does not survive staff turnover.

Solution Overview – “PushCheck”

“PushCheck” is an AI‑driven diagnostic expert that replaces blind troubleshooting with a transparent, traceable, and predictable workflow. It consists of two complementary execution tracks:

Highway : a deterministic, script‑driven pipeline handling ~80% of common issues by matching incoming cases to pre‑defined Story scripts.

ATV : an autonomous ReAct‑based agent tackling the remaining ~20% of long‑tail, complex anomalies that Highway scripts cannot resolve.

A central Intelligent Scheduler routes each request: if a knowledge‑base lookup matches a standard Story, Highway executes; otherwise, the system falls back to ATV.

OpenClaw Plugin Architecture

The team evaluated generic open‑source frameworks versus the proprietary OpenClaw and chose the latter for its deterministic governance. OpenClaw introduces:

Atomic Skills : encapsulated API calls (plugins) that can be composed into Stories, enabling decoupled, version‑controlled diagnostics.

Deterministic Controls : iteration limits, tool whitelists, and confidence thresholds that prevent infinite loops and hallucinations.

Ecosystem Compatibility : Skills can be reused across other internal AI scenarios.

Highway & ATV Dual‑Mode

Highway executes a fixed sequence of Steps defined in a Story, for example:

Story: 商品推荐异常-标准诊断
├── Step 1: screenshot‑trace‑query   # 截图识别，自动匹配Trace
├── Step 2: user‑profile‑query      # 查用户购买/点击历史
├── Step 3: layer‑items‑query      # 看商品在各层曝光情况
├── Step 4: strategy‑logs‑query   # 核对策略干预（去重/过滤）
├── Step 5: commodity‑forward‑query # 查商品正排信息
└── Step 6: output_diagnosis       # 输出诊断结论

ATV follows the classic ReAct loop:

Question → Thought → Action(Skill) → Observation → Final Answer

To keep ATV safe, three constraints are enforced: maximum six reasoning iterations, a tool whitelist, and a confidence‑threshold check that forces a fallback when the model is uncertain.

Knowledge‑Base Integration

Because large‑scale trace logs and documentation exceed token limits, the system combines two components:

OpenViking : a local context engine that stores memory, handles multi‑Agent isolation, and performs semantic retrieval.

Graphify (GitHub: https://github.com/safishamsi/graphify): transforms files into entity‑relationship graphs, drastically reducing token consumption and improving code‑document understanding.

The combined stack enables fast, token‑efficient retrieval of technical documents and supports multi‑Agent collaboration.

Evolution Layer – From Records to Methodology

The evolution pipeline consists of four steps that automatically distill raw Agent traces into reusable Stories and Skills:

Trace Pruning & Efficiency Extraction : replay past cases, keep only steps that decisively contributed to root‑cause identification, reducing a 10‑step process to 3‑4 steps.

Feature Upscaling & Logic Generalization : map concrete identifiers (UIDs, RequestIDs) to high‑dimensional business variables, ensuring portability across users and time.

Story & Skill Generation : emit structured skill.md files containing trigger conditions, execution logic, and assertions.

Sandbox Dry‑Run & Admission Check : replay the new Story/Skill against the last 7 days of real cases; if automated conclusions match expert verdicts >90%, the artifact is promoted to Highway, otherwise it returns to the evolution layer.

Real‑World Case: Duplicate Product Recommendation

A production case where a user repeatedly received phone‑case recommendations was processed automatically. The generated Story executed the six steps above and produced the following diagnosis:

Item was recalled via the TIDE channel.

Recall source stored in item.extMap.cns_v2.

User had recent phone‑case browsing signals, triggering normal recall.

Conclusion: not a bug, but over‑exposure affecting user experience.

Final Reflections

“PushCheck” transforms the recommendation system from a black‑box debugging arena into an AI‑augmented, self‑evolving diagnostic platform. It preserves deterministic guarantees while allowing autonomous reasoning for edge cases, continuously learning new Skills from resolved incidents. The authors envision extending the system from diagnosis to proactive prevention, paving a path toward AI‑for‑systems and eventual autonomous operations.