Scaling Intelligent Q&A for Internationalized IC Using Spring AI Alibaba

Background

The Item Center (IC) receives more than 20 developer queries per day—trace error analysis, error‑code lookup, tag usage analysis, product visibility and salability diagnosis, change‑record tracing, and data queries—with an average resolution time of 30 minutes. The manual Q&A process is inefficient and costly.

Limitations of the Existing Agent

Only single‑turn Q&A no multi‑user, multi‑turn dialogue, causing context loss in group chats.

Single‑route decision; a question can be routed to only one Agent, unable to handle composite queries.

Rough document splitting; recall accuracy is low and the knowledge‑base quality score is only 8.0.

No Skills support; adding a new capability requires creating a new Agent, leading to a long iteration cycle.

Inefficient MCP component packaging; each component maps to a separate interface, making reuse of mature scripts difficult.

Upgrade Goals

Using the Spring AI Alibaba framework, the team rebuilt the Agent architecture to support multi‑user, multi‑turn conversations, integrate Skills & MCP, enable hot‑updates of Skills, and raise the automated answer‑rate from 60 % to over 80 %.

Framework Selection

Spring AI Alibaba offers the highest abstraction level (Spring AI > LangGraph > LangChain) and provides out‑of‑the‑box Graph orchestration, ReAct Agent, ChatMemory, and ToolCallback components, accelerating development compared with LangChain (low‑level assembly) and LangGraph (Python‑centric).

The official demo integrates easily with internal middleware such as Aone memory service, MetaQ messaging, and ScheduleX scheduling, reducing integration effort.

Architecture Overview

Access Layer : DingTalk group‑chat bot receives messages via im.chat.message and replies with card‑style streaming output via robot.send.

Orchestration Layer (Spring AI Alibaba Graph) executes a standardized sequence of nodes: ConversationContextNode: retrieves historical messages from Aone memory and trims context according to token limits. IntentClassificationNode: lightweight model classifies intent and routes knowledge‑base queries to the RAG chain and execution tasks to the Agent chain. ExecuteAgentNode: runs the ReAct loop (Thought → Action → Observation) and dynamically invokes Skills and MCP tools. ConversationSaveNode: persists the current turn.

Execution Layer : ReAct reasoning core; Skills provide domain‑specific capabilities (trace analysis, error‑code lookup, etc.), MCP provides generic tool calls (script execution, code search). The Agent decides which Skill/MCP to invoke at runtime.

Infrastructure Layer : custom RAG knowledge base (self‑designed document splitting + rerank), Aone memory service, AgentTrack SDK, Switch, Diamond config center, MetaQ queue, and other foundational services.

Detailed Design

Dialogue Capability

Before : No user ID in group chat, causing loss of context for follow‑up questions.

After : Messages include a (group‑ID + user‑ID) tuple; session state is stored in Aone memory with a sliding window that keeps the last five rounds. The system supports more than five rounds of parallel multi‑user dialogue with zero cross‑talk.

RAG Capability

Before : Direct import of Yuque documents without preprocessing; unstable splitting leads to low recall.

After : Document preprocessing via Qoderworker and Aone Copilot using prompt‑driven cleaning. A custom MD hybrid splitter works in two stages:

Stage 1 – a single‑pass state machine parses Markdown into block‑level elements and builds a title‑path stack (e.g., “活动管理 > 独占模式”), assigning each block its hierarchical context.

Stage 2 – “small‑block aggregation, structure protection, long‑text semantic split”: tables and code blocks are never split, adjacent short paragraphs are merged, and long text blocks are split by semantic vectors to keep each chunk semantically complete.

Each chunk carries title‑path and content‑type metadata, which significantly improves RAG recall. Retrieval uses a dual‑path strategy—semantic similarity + keyword search—followed by a Cross‑Encoder rerank to correct coarse‑ranking bias.

Execution Capability

Before : Five independent sub‑Agents, each with a single route; adding new abilities required a new Agent.

After : The five sub‑Agents are standardized into six Skills defined in SKILL.md plus a product‑query Skill. Skills can be hot‑updated via Diamond configuration changes, MetaQ events, or ScheduleX fallback. Aone MCP and a custom Toolbox MCP are integrated; the Agent autonomously selects tools at runtime.

Example Skill URLs:

Trace error analysis:

https://open.aone.alibaba-inc.com/console/platform/aidc-dev/skill/i18n-ic-trace-analyzer

Error‑code analysis:

https://open.aone.alibaba-inc.com/console/platform/aidc-dev/skill/i18n-ic-error-code-analyzer?tab=readme

Tag usage analysis:

https://open.aone.alibaba-inc.com/console/platform/aidc-dev/skill/i18n-ic-tag-usage-analyzer

Change‑trace diagnosis:

https://open.aone.alibaba-inc.com/console/platform/aidc-dev/skill/i18n-ic-product-change-traceability-analyser?tab=publish

Salability analysis:

https://open.aone.alibaba-inc.com/console/platform/aidc-dev/skill/i18n-ic-product-salability-analyzer?tab=publish

Product information query:

https://open.aone.alibaba-inc.com/console/platform/aidc-dev/skill/ic-ae-product-query

Model Selection

Before : A single large model handled all tasks, resulting in high cost and latency.

After : Two‑stage model strategy:

Intent classification uses the lightweight qwen‑flash model (fast, low cost).

Agent reasoning and tool calling use the powerful qwen3‑max model (strong function‑calling, stable output).

Comparison dimensions:

First‑token latency – lower for qwen‑flash, higher for qwen3‑max.

Function‑calling – not applicable for qwen‑flash; qwen3‑max provides stable, high‑accuracy calls.

Inference cost – low for qwen‑flash, higher for qwen3‑max.

Applicable scenarios – lightweight tasks vs complex reasoning & tool use.

Evaluation System

The team built a four‑pillar evaluation framework covering system runtime metrics, RAG recall quality, specialized capability tests, and end‑to‑end Q&A experience.

Metrics

Latency (client‑side end‑to‑end time)

TokenUsage (cost per turn)

TTFT (time to first token)

RAG Recall Metrics

Hit Rate@K, Recall@K, Precision@K, MRR, NDCG@K

Average Relevance Score

Effective Retrieval Rate

Zero Result Rate

Average Latency

Specialized Capability Metrics (1‑10 scale)

QA_SolutionRate, QA_Relevance, QA_Consistency, QA_Accuracy, QA_Readability

Evaluation Execution

Design test sets for each capability (trace analysis, error‑code lookup, tag usage, change‑trace, salability, product query).

Generate realistic queries from historical tickets and LLM‑generated samples; manually verify ground‑truth answers.

Run the target Agent version against the test set, store results, and auto‑score via an evaluator Agent.

Results

Basic Skills (Specialized): 50 cases, 49 passed → 98.0 % pass rate.

Knowledge‑Base Q&A (General): 130 cases, 110 passed → 84.6 % pass rate.

Abnormal Input Handling: 19 cases, 19 passed → 100 % pass rate.

Multi‑Turn Dialogue: 10 cases, 5 passed → 50 % pass rate.

End‑to‑End Comprehensive: 209 cases, 179 passed → 85.6 % pass rate.

Total: 418 cases, 362 passed → 86.6 % overall pass rate.

Highlights

Specialized capabilities reached 98 % pass; trace analysis, tag usage, change‑trace, and salability all achieved 100 %.

Intent classification recall is 100 %, ensuring no query is dropped.

All 19 abnormal input types (empty, garbled, SQL‑inject, XSS, emojis, long repeats, etc.) were correctly routed to a safe fallback with no security breach.

Areas for Improvement

Faithfulness (hallucination) rate is 69‑71 %; plan to add stronger prompts, citation enforcement, and refusal strategies.

Multi‑turn success rate only 55.6 % (absolute 50 %); context propagation across turns needs reinforcement.

Domain Q&A fidelity is 82.2 %; long‑tail documents still produce fragmented chunks, requiring more manual split verification and higher rerank weight.

First‑token latency is high (overall 19.9 s, specialized 27.1 s); will shorten context windows and optimize memory injection.

Future Outlook

Channel expansion: integrate ticket systems, internal “XiaoMi” bot, etc.

User profiling: embed user persona into session context for more targeted answers.

Risk‑control Skills: inject detection and sensitive‑info masking.

Long‑term memory: persist high‑quality answers and user context to enrich the knowledge base.

RAG document enhancement: multi‑source retrieval from code repositories and Wiki.

Extend operational Skills: write‑operations, tag‑management, access‑request handling.

Conclusion

The migration from isolated single‑turn Agents to a standardized Spring AI Alibaba framework delivered substantial gains in dialogue depth, knowledge coverage, and extensibility. The Skills & MCP hot‑update mechanism reduced capability rollout from days to hours, while the custom RAG pipeline and automated evaluation system made knowledge quality measurable and iteratable. Ongoing work will focus on broader channel reach, richer user context, stronger safety, and continual memory growth to evolve the system from merely usable to truly delightful.