Building an AI Q&A Agent for a Global Commodity Center: Practice, Frameworks, and Evaluation

1. Introduction

Alibaba International Commodity Center (IC) manages massive product information, permissions, tags, and cross‑system coordination. Rapid growth in product volume and scenario complexity made traditional document‑based, manual support inefficient, prompting the creation of an AI‑driven intelligent Q&A Agent built on the App Builder platform.

2. Motivation

Daily queries from operators, developers, and product teams include:

“What is the cause of error code XXX?”

“Why is product XXX not visible?”

“What does tag XXX do?”

“Which offline table contains field XXX of product?”

“Why does product information differ from order data?”

These questions span error‑code analysis, visibility, tag usage, offline data, and change‑record diagnostics, leading to high manual investigation cost and long latency.

3. Standardized Framework Construction

The team defined a three‑stage technical path to evolve from a single‑agent to a multi‑agent collaborative framework.

3.1 First Stage – Single‑Agent Framework (Scene Identification + Tool Invocation + SOP Execution)

The single‑agent consists of four modules:

Perception Module : receives user input, uses a scene knowledge base to classify the scenario, extract keywords, and identify intent. Provides guidance and fallback strategies.

Planning Module : based on perception results, selects tools, defines execution order, and formats the final output using an SOP knowledge base.

Tool Module : executes scripts (via the ToolBox platform) or MCP protocol calls; tool definitions are stored in a tool knowledge base.

Memory Module : short‑term memory holds the current session (user input, prompts, tool results); long‑term memory connects to external RAG knowledge bases.

Issues identified:

Heavy functional coupling : changes in one scenario affect others; responsibilities are blurred.

Long response latency : tool results can be large, causing context overflow and truncation.

High maintenance cost : version mismatches and extensive testing are required for new capabilities.

3.2 Second Stage – Multi‑Agent Fusion Framework (Scene Identification + Single Routing)

A master control Agent performs intent recognition using the scene knowledge base. If the intent is clear, it routes the request to a specialized Agent (e.g., product visibility analysis). If not, it falls back to the IC Document Query Agent for generic knowledge retrieval.

Remaining challenges:

Only a single scenario can be addressed per request, limiting coverage of composite queries.

Agents do not share intermediate states, preventing collaborative reasoning.

Complex decision flows remain hard to express.

3.3 Third Stage – Multi‑Agent Collaboration Framework (Intent Decomposition + Multi‑Scene Routing)

To handle cross‑scenario problems, the system decomposes user intent into multiple sub‑intents, dispatches each to the appropriate specialized Agent, and aggregates the results. Example: for “Why is product X unsellable?” the system first runs a visibility analysis Agent, then, if the cause is an audit status, invokes the change‑record diagnostic Agent to provide root‑cause details.

4. Core Module Details

4.1 Perception Module Prompt

# 角色设定
你是一位经验丰富的国际化商品中心（IC）技术专家，熟悉各种场景的用户咨询问题，擅长分析用户问题，识别用户问题的场景；
# 专业技能
1. 分析用户问题，结合#商品AI助手-场景知识库中的question_example和keywords_example两列数据，自主提取用户问题中的关键词并进行精准匹配，返回知识库命中行的所有数据。
2. 进行适当联想和问题归类，确保匹配的准确性。
# 输出限制
1. 严格按照#商品AI助手-场景知识库中定义的场景进行匹配。
2. 若无法匹配，则scene_code设为`default`。
# 输出格式示例
## 匹配到知识库对应场景
```json
{
  "user_request": "${用户问题}",
  "scene_code": "${scene_code}",
  "scene_name": "${scene_name}",
  "keywords": "${用户问题关键词}"
}
```

4.2 Tool Module

Tools are defined in a tool knowledge base with input/output formats. Representative tools include:

代码搜索 (MCP) : semantic search of main‑branch code in the repository.

语雀文档关键词检索 (MCP) : general document search and detail retrieval.

商品中心请求摘要日志查询 (脚本工具) : site‑wide service_monitor log query (up to 60 days).

商品中心出入参日志查询 (脚本工具) : site‑wide app_trace log query (up to 60 days).

商品中心变更日志查询 (脚本工具) : change‑log query (up to 60 days).

标签元数据查询 (脚本工具) : retrieve tag metadata by tag code.

商品可见可售feature查询 (脚本工具) : fetch per‑country visibility/sale feature for a product/SKU.

商品可见可售feature解析 (脚本工具) : decode the feature.

商品关键字段变更记录查询 (脚本工具) : query change records for product/SKU fields.

4.3 Memory Module

Short‑term memory stores user input, prompts, scene description, and tool results for the current session. Long‑term memory integrates external RAG knowledge bases and can be injected selectively during orchestration.

5. Evaluation System

A comprehensive evaluation pipeline was built to make Agent capabilities measurable and iteratable.

5.1 Evaluation Design

Two test sets were created:

Specialized scenarios : 5 Agent types (trace tracing, error‑code analysis, tag usage, visibility analysis, change‑record diagnosis) with real tickets and manually verified answers (50 cases).

General scenarios : 7 question types covering basic info, database tables, offline data, access control, pricing models, change messages, and HSF interfaces, generated with Claude‑4.5‑Opus and manually corrected (140 cases).

5.2 Evaluation Metrics

Five dimensions were scored (0‑10):

Question‑Solution Rate (QA_SolutionRate)

Task Relevance (QA_Relevance)

Logical Consistency (QA_Consistency)

Information Accuracy (QA_Accuracy)

Readability (QA_Readability)

5.3 Evaluation Execution

Using the AE Quality Evaluation Platform, test cases were stored, the Agent version under test was invoked via HTTP, and a GPT‑5‑based evaluator automatically scored each response.

5.4 Results

Specialized scenarios (50 cases) – average overall score 8.73:

Relevance 9.29, Consistency 9.23 (highest).

Solution rate 8.79, Accuracy 8.41, Readability 7.93.

Shows strong domain expertise but room for factual precision and language polish.

General scenarios (140 cases) – average overall score 8.29:

Consistency 9.11, Readability 8.40.

Relevance 8.20, Solution rate 7.90, Accuracy 7.82.

Good reasoning but weaker factual correctness for broader queries.

6. Effect Demonstration

Sample outputs for each specialized Agent (trace tracing, error‑code analysis, tag usage, visibility analysis, change‑record diagnosis) and for general scenarios (basic info, database tables, offline data, access control, pricing models, change messages, HSF interfaces) are shown as screenshots.

7. Future Outlook

Upgrade to the intent‑decomposition + multi‑scene routing collaborative framework to support complex, cross‑scenario queries.

Continuously enrich the document knowledge base and improve semantic retrieval.

Build an Atlas “click‑to‑diagnose” front‑end that unifies core specialized Agent capabilities into an intelligent workbench.

8. Core Diagrams

Architecture of the three stages and the evaluation pipeline are illustrated in the following images: