From ChatBI to DataAgent: Turning AI Demos into Trusted Enterprise Decision Engines

Background and Motivation

The DataFunSummit livestream on April 9 brought together a host and three experts—Yang Zhouzhi from Xiaohongshu’s data analytics platform, Yan Lingang from Guanyuan Data, and moderator Du Shujun—to explore why the gap between a demo‑level AI and a production‑ready Data Agent is not about model strength but about semantic convergence, knowledge structuring, system explainability, and business trust.

1. Unified Semantic Layer: Beyond a Concept

Participants debated whether the semantic layer should be implemented as a Cube, a View, or an API abstraction, emphasizing that the layer is the foundation for accurate data retrieval. Yang highlighted the difficulty of mapping user‑facing dimension values to stored values, describing Xiaohongshu’s approach of extracting high‑frequency dimension values into an acceleration engine and using semantic understanding to map natural language back to correct data objects. Yan added that the semantic layer can take various forms—To‑SQL, To‑DSL, or intent‑driven front‑ends—depending on the scenario, and that many errors stem from mis‑interpreted business jargon rather than SQL generation.

2. Prompt Engineering vs. Super‑Fine‑Tuning (SFT)

Yang explained that prompt engineering yields quick gains (60‑70% accuracy) but quickly hits diminishing returns, prompting a shift to SFT to push accuracy to around 85%. Yan argued that heavy model fine‑tuning is often uneconomical for multi‑client products; instead, they maintain stable system prompts, use benchmark regression tests, and inject knowledge modules (error cases, SQL constraints, analysis methods) dynamically during inference.

3. Table Discovery: From “Finding Numbers” to “Finding the Right Table”

Yang’s team first narrows the data scope by business line, then performs intent recognition, candidate matching, and semantic tagging, distinguishing tables suited for detail versus aggregation. Yan stressed that effective table discovery relies on solid data governance—clear annotations, usage metrics, and thematic data spaces—so that the Agent can treat assets as trusted knowledge.

4. Consistency of Metrics: Who Should the Agent Listen To?

Both experts agreed that metric definitions must be centralized in a unified KPI platform; the Agent should not invent definitions but defer to the organization’s approved taxonomy, with routing mechanisms to select the appropriate data source for a given query.

5. Multi‑Turn Conversation: Remembering What Matters

Yang described a two‑tier memory system: short‑term memory compressed within the context window and long‑term memory built from offline summaries of user habits. Yan prefers trimming conversation turns and rewriting questions to retain only essential information, avoiding token bloat while preserving critical constraints.

6. Trust and Explainability

Explainability is essential for business trust. Yang proposes exposing the reasoning chain and linking results back to underlying data configurations, dimensions, and filters. Yan implements parallel result generation and cross‑validation, and ensures that final reports are populated by scripts that fetch data directly from sources rather than relying on model‑generated numbers, thus minimizing hallucinations.

7. Cost Management When Model API Prices Surge

Yang would first cut high‑token collaborative Agent scenarios, explore distilling models or local deployment, but retain semantic caching to reduce latency and cost. Yan suggests applying expensive models only to high‑value decision meetings, while routine reporting can rely on rule‑based or templated pipelines.

8. Continuous Improvement: SFT and RAG Flywheels

Yang outlined an SFT flywheel: cluster online questions, select representative samples, feed them into a training pool, perform AI pre‑screening and human review, then fine‑tune and redeploy. Yan described a three‑layer flywheel: user feedback, Bad‑Case/Prompt optimization, and industry‑wide knowledge assets that feed back into the platform.

9. DSL vs. SQL Debate

Yang favors a DSL on top of the semantic layer for better control in enterprise BI, while Yan remains neutral, noting that the choice between SQL, DSL, or direct dashboard queries depends on robust intent recognition and context understanding.

Conclusion

The panel concluded that the true watershed from ChatBI to Data Agent is not a stronger model but a system that earns business trust through reliable semantics, governance, explainability, and sustainable operational loops, effectively turning the Agent into a trustworthy analytical colleague rather than a generic chatbot.