Xiaomi Data Agent: Why Engineering Harness Beats Model Alone for Production Analytics

Why a Data Agent Despite a 50k-WAU BI Platform

Xiaomi already operated a BI platform with 50,000 weekly active users, but BI faced four walls: high professional barrier (SQL, dashboard building), lack of intelligence (only shows numbers, root-cause still needs analysts), poor flexibility (new metrics require scheduling), and broken decision loops (action stops after viewing data). The Analysis Agent is not a BI replacement but an enhancement layer covering natural-language querying, root-cause analysis, flexible drill-down, and rule-to-action closure.

Product Form: Consumption Mode & Configuration Mode

Consumption mode (user-facing) covers "Query → Analyze → Present": conversational querying with multi-turn follow-up, quick query, human-in-the-loop, and Skill invocation; intelligent analysis provides anomaly detection, root-cause diagnosis, and diagnostic suggestions; trusted presentation uses report review, data lineage, and HTML visualization for auditable conclusions.

Configuration mode (admin-facing) covers "Access → Evaluate → Tune → Launch → Monitor": data and semantic onboarding (import, semantic management, knowledge config); Agent tuning loop (Agent config, evaluation module, auto-iteration); post-launch row/column permissions, online monitoring, and multi-channel access.

Results: deployed across 10+ business domains , sustaining 1,000+ weekly active users , producing thousands of analysis reports and insights, with 10x+ efficiency gains in data usage, analysis, and decision-making. Externally, the team ranked 3rd globally on the BIRD Text-to-SQL leaderboard , but the leaderboard only tests "can solve", while production demands "stably correct over time".

From "Can Answer" to "Continuously Answer Correctly": Five Production Gaps

① Data Quality | Multi-Level Semantic Fallback: Metric → DSL → Reject

Poor data quality amplifies model hallucinations. The system uses a three-layer fallback:

Metric layer (highest trust) : Text2JSON maps user questions to certified, continuously maintained atomic, derived, and composite metrics.

Semantic model layer : Text2DSL + built-in parser maps measures, dimensions, relationships, entities to DSL, handling JOIN paths, granularity, GROUP BY, and row/column permissions.

Base-table layer : Direct Text2SQL on ODS/DWD/DWS/ADS, requiring manual verification.

Reject : If no valid query can be formed, the system refuses to answer. Core principle: "Better not answer than answer wrongly without a trusted source." Multi-layer design also accommodates varying semantic maturity across businesses.

Existing BI semantic models cannot be handed directly to LLMs. The system analyzes schema, production jobs, code, query history, data lineage, BI components, and metric definitions to compute table/field features, user behavior, and metric caliber. A Diagnostic Suggestion Agent scores quality and proposes completions; users can accept, reject, modify, or customize. Only when quality score exceeds 80 is QA enabled.

② Generation Stability | Multi-Path Generation + Voting

Even with low temperature, the same query can produce minor variations across batches or MoE routing; complex SQL amplifies drift. Solution: multi-path generation with voting .

First two phases: data understanding and semantic understanding (AI diagnostic enhancement, dimension-value dictionaries, business docs, rewriting, tokenization, multi-path recall ranking, Schema Link, Verified SQL, skeleton Few-shot).

Generation phase: multiple models and strategies produce candidate SQLs; SQL Fixer retries up to 3 times on errors or empty results; async concurrency, timeout discard, and prompt cache control latency.

Selection phase: cluster by execution results; confidence = size of largest cluster / total candidates. Confidence ≥ 0.6 → take largest cluster SQL ; low confidence enters Pairwise discrimination comparing top-2 clusters with multi-run self-consistency. Offline greedy search finds optimal LLM+strategy combos.

Backtesting shows variance converged from ±5% to ±1% . This mode is opt-in because it trades extra inference cost for stability.

③ Generation Trust | Explainability = Trust: Data Lineage

An analysis report may involve multiple tool calls with non-unique paths. Instead of forcing identical reports, the system ensures every number is traceable . Sources are classified as:

Tool invocation : save raw data and context; write claim_id during report generation.

LLM computation : parse aggregation, filtering, JOIN, derived formulas from SQL, root-cause analysis, or Python results.

LLM deduction : numbers that cannot be reverse-matched (estimates, predictions) are marked non-traceable and downgraded .

Final confidence (high/medium/low/non-traceable) is based on source type, match score, and ambiguity; users can drill down to origins.

④ Evaluation Construction | Cold Start: BI Dashboards + Intelligent Generation

Pre-launch evaluation sets are mandatory but manual Query-SQL authoring is costly. Existing BI dashboards serve as cold-start assets : the system traverses components, converts descriptors to DSL, verifies data consistency; a Dashboard Analysis Agent combines global filters, component filters, titles, and user habits to generate natural-language queries, forming QA pairs for human review and storage. Without dashboards, batch generation from semantic models and table fields works: users configure common metrics, dimensions, time ranges, question styles, and sample counts, then inject business knowledge.

Post-launch, real queries continuously feed the evaluation system. Thumbs-up = strong positive signal; thumbs-down or error = strong negative signal ; multi-turn, empty results = weak signals. AI only proposes error causes, SQL drafts, and knowledge suggestions; humans confirm samples, SQL, and knowledge before persistence. Goal: turn "write from scratch" into "review drafts" .

⑤ Optimization Efficiency | Loop: Evaluate → Attribute → Enhance → Re-evaluate

With an evaluation set, bad cases are continuously discovered. The "Evaluate → Analyze → Patch Semantics → Re-evaluate" loop is codified :

Full evaluation uses Text2SQL + Judge.

Error Classification Agent categorizes failures: table selection, SQL syntax, SQL logic, semantic understanding, result deviation.

Routing follows human experience for fixes.

Review Agent checks generalization, conflicts with existing knowledge, and new errors; on pass, changes are applied, version snapshot saved, and re-evaluation triggered.

Stops when target met, bottleneck hit, or regression detected (then rollback).

Users just one-click start optimization and review changes via Feishu notification.

⑥ Anti-Decay | Backtest Monitoring + Incremental Accumulation + Semantic Layer CI

Hitting 90% accuracy and launching is not the end. Three defense lines:

Periodic automatic backtests with alerts on degradation.

Continuous ingestion of real queries into incremental evaluation sets .

Semantic layer changes go through CI like code , with automatic before/after evaluation comparison.

Launch is just the starting point; continuous decay is the norm for data systems.

Three Real Scenarios: How Data Agent Enters Business

After launch: quick query from 1 hour–1 day down to 17 seconds ; root-cause analysis from 1–3 days to 2–5 minutes ; analysis reports from 1 week to 10 minutes ; new business onboarding from month-level to day-level . Core value: freeing people from repetitive data pulling.

Financial Sales: Dual Needs — Document Q&A and Data Q&A

RAG document Q&A with multi-turn clarification for documents; Verified SQL, multi-path voting, and numeric traceability for data; intent recognition auto-routes, also supports RAG+SQL composite queries. Accuracy held at 90%+, efficiency up 20x+, with 800+ evaluation cases under continuous operation.

Human Resources: Non-Technical Users, Diverse Templates

Same data analyzed by different dimensions (campus hires, middle managers, blue/white collar) with varying report templates. Skills encapsulate analysis logic and visualization templates ; HTML enables rapid switching between detailed, concise, monthly styles. 200+ MAU, 3.2 conversations per user per week ; non-technical HR self-serve data and reports.

Bill Analysis: From Cost to Insight to Action

Focus not just on "how much more spent" but "why" and "who owns it". Numeric traceability and anomaly root-cause diagnosis turn data into insight ; then "Generate report → Find owner → Sign off → Implement fix → Verify closure" pushes insight to action . Report generation from 3 days to 10 minutes , covering 10+ domains.

New Business Onboarding as Standard SOP

Platform side: deployment-delivery FDE provides platform capabilities. Business side: data-savvy business expert FDE collaborates. Data access, evaluation construction, effect optimization each take ~2–3 days; launch iteration continues. Overall 1–2 weeks to onboard a new business.

After Everyone Can Build Agents, What Should the Middle Platform Do?

Models alone are insufficient; reliability comes from the Harness engineering loop. On the same BIRD leaderboard, the engineering harness ranked 3rd while pure model baselines mostly sat at 30–90+ . Data analysis differs from code generation: context data quality is worse; many questions have near-unique answers; verification must go beyond SQL executability because business caliber lives in human knowledge and semantic systems.

The barrier to launching an Analysis Agent has dropped sharply. Metadata acquisition and data query interfaces exposed via CLI mean teams need not build full infrastructure; vertical teams focus on their own data (smaller table scale, deeper domain knowledge); tools like Claude, Codex and Skills mechanism further lower dev and knowledge-reuse costs. Hence every business team can rapidly build its own vertical Analysis Agent.

Low barrier ≠ low long-term maintenance cost. The middle platform should centrally solve three problems:

Build semantic-layer governance infrastructure : standardize semantic structure, business knowledge import, CI, versioning, and lineage impact analysis.

Evolve the Harness continuously : keep lowering evaluation, backtest, monitoring, and business onboarding costs.

Connect Development, Governance, and Analysis Agents , and open CLI and Skills to fill end-to-end capabilities that single business teams cannot build alone.

From the middle-platform perspective, the focus is not doing what others cannot, but turning what is high-cost, long-maintenance, and low-efficiency for business teams into reusable platform capabilities .