Turning R&D Support into Scalable AI Agents: A Blueprint for Operable Knowledge

Background and Pain Points

Engineers spend a large portion of their time switching between tickets, chat groups, public opinion feeds, and manual troubleshooting, which leads to low response efficiency, unstable handling quality, and steep onboarding curves for newcomers.

Root causes include knowledge fragmentation, lack of systematic knowledge storage, and complex, multi‑service architectures that make debugging feel like a "Long March".

Problem Abstraction

All high‑frequency support requests can be grouped into two core capability types:

Business Q&A (explanatory capability) : answer questions such as "Why does user see B while I see A?" or "What does this field mean?" The output must contain rule explanations, reference links, and clear conclusions.

Issue Diagnosis (diagnostic capability) : resolve incidents like "Order status stuck" or "Refund amount mismatch". The output must include evidence chains, step‑by‑step troubleshooting, and actionable remediation.

Both capabilities share a common output schema:

Conclusion + Analysis (rules) + Optional troubleshooting plan + Action suggestions + Document references

Four‑Layer System Architecture

Channel Layer : ticket systems, opinion platforms, IM groups, partner portals. Handles multi‑modal input preprocessing.

Orchestration Layer : intent recognition (explanatory vs diagnostic) and routing to the appropriate agent.

Agent Layer : LLM, Retrieval‑Augmented Generation (RAG), diagnostic tools, knowledge base, context assembly, and tool‑calling strategies.

Ops & Eval Layer : answer management, follow‑up tracking, quality scoring, monitoring dashboards, and feedback closure.

Design Principles and Evolution

The system must be persistable, reusable, evaluable, and easily iterable . Based on this, the architecture was refined through three major implementation patterns:

1. Java‑Driven Flow (Initial Mode)

All troubleshooting steps are hard‑coded in Java and invoked via predefined tools. While stable, this approach suffers from low flexibility and high maintenance cost.

2. Prompt‑Embedded SOP (Intermediate Mode)

Standard Operating Procedures (SOP) are written directly into prompts, allowing the model to follow explicit step‑by‑step instructions. This improves flexibility but leads to prompt bloat and context pollution.

3. Workflow Mode

Separate workflow engines orchestrate atomic tool calls. Adding new capabilities still requires workflow redesign, limiting rapid iteration.

To overcome these limitations, a new paradigm was introduced: treat each diagnostic document as a skill closure . Adding a capability now means merely authoring a new SOP document, without touching prompts or code.

Document template example (simplified):

# Applicable Scope
Brief description and usage example

# Field Explanation (optional)
Explain key fields, e.g., pjyl = 1 means reward issued

# Core Log Format (optional)
Guidelines to avoid hallucination

# Diagnosis Steps
- Step 1: Retrieve evaluation details
- Step 2: Check reward status
- Step 3: Verify ttid
- ...

Quality Evaluation and Feedback Loop

Traditional IR metrics (Recall, Precision, F1) are insufficient for multi‑step support answers. A fine‑grained Q‑score (0‑10) was introduced, evaluating completeness, correctness, adherence to SOP, and usefulness. Scores ≥7 are considered "effective answers".

✅ Effective answers reduce manual intervention while still delivering production‑ready value.

An automated scoring agent consumes the interaction history and knowledge‑base state, applies few‑shot examples and domain rules, and outputs a Q‑score with detailed penalty items (e.g., skipping steps, using outdated docs).

The scoring results feed back into a closed‑loop process:

Online Q&A → Scoring Agent → Focus low‑score samples (≤6) → Human review → Root‑cause analysis → Update SOP docs / public knowledge / prompts → Retrain → Loop

This loop accelerates knowledge consolidation, drives model behavior convergence, and provides transparent operational monitoring.

Operational Metrics and Case Studies

Deployed across multiple domains (order management, logistics, merchant Q&A, evaluation, reverse‑flow diagnosis), the agent system achieved:

Recall and Precision often above 80% per domain.

Average Q‑score above 7, indicating high answer quality.

Significant reduction in manual ticket handling time.

Representative case studies include:

Evaluation domain: automated detection of reward‑issue anomalies, reducing manual checks.

Reverse‑flow diagnosis: one‑click reset of data‑link problems without developer involvement.

Merchant Q&A: rapid resolution of order‑related queries.

Future Directions

Key next steps focus on lowering the barrier for skill creation and enhancing real‑time feedback:

Automatically generate initial SOP drafts from code annotations, API docs, and link logs, followed by expert review.

Implement minute‑level anomaly detection and auto‑alerting to replace monthly manual reviews.

Explore "AI‑native" knowledge organization where models directly produce executable semantic instructions, shifting knowledge creation from human‑first to AI‑first.

Ultimately, the goal is to let newcomers handle complex incidents as confidently as veterans, achieving genuine engineering productivity gains.