From Business Question to Traceable Report: How DataSage Builds Reliable Data Analysis Agents

DataSage (Ant DataSage) is a next-generation data analysis agent developed by Ant Group's Intelligent Analysis Platform team. It provides end-to-end analysis: understanding requirements, planning paths, decomposing tasks, executing via tools, and delivering tables, root-cause conclusions, and full reports. The goal is to let AI autonomously drive the complete analysis loop while retaining user confirmation, process tracing, and result acceptance.

Core Pain Points in General-Purpose Agents for Data Analysis

A typical request — "Check last week's GMV changes; if anomalies exist, drill down on core dimensions and push a report" — implies a chain of five steps: (1) confirm metric definition (payment vs. order, refunds inclusion, granularity, scope), (2) baseline comparison and anomaly detection (historical, week-over-week, trend, business thresholds), (3) trigger multi-dimensional drill-down (channel, category, region, segment, campaign, price band) with contribution and significance evaluation, (4) produce an auditable report recording definitions, paths, tables, charts, and derivation logic, (5) deliver to DingTalk with support for follow-up and reuse.

General-purpose agents exhibit four systemic failures on such long-running, multi-turn, data-accountable workflows:

Requirement understanding and orchestration drift: models skip definition confirmation, jump to conclusions, or choose mismatched analysis steps.

Tool expansion becomes non-standard and inefficient: adding DB, Python, Excel, search, doc, and push tools inflates prompt size and orchestration cost; tool count growth does not proportionally increase task completion.

Massive data context degrades long/multi-turn quality: accumulated query results, intermediate calculations, charts, user corrections, and tool traces cause token overflow, attention dilution, and semantic drift.

Opaque definitions and lineage undermine trust: a successful tool call only proves local execution, not correct metric definition, complete calculation scope, chart-data consistency, or traceable conclusions.

DataSage Harness: Four Design Pillars

Models excel at language understanding, hypothesis generation, path organization, and result interpretation. Systems should handle permissions, state management, persistence, deterministic computation, evidence verification, and delivery confirmation. DataSage Harness addresses the four pain points through:

Analytic Core Engine: Agentic Loop + Data Planning — constrains understanding and execution entry via analysis plans and state machines.

Standardized Pluggable Multi-Agent Tool System — organizes domain capabilities via Agent-as-Tool.

Massive Data Context Management — uses layered context and data summaries to control information load.

Traceable Calculation Definitions and Evidence Chains — defines true completion via artifact acceptance, evidence citation, and delivery receipt.

Evolution of the Analytic Core Engine

Phase 1: ReWOO Planner for Decomposition and Planning

Inspired by ReWOO (plan-then-execute-then-solve), DataSage initially fused domain expert experience into a ReWOO Planner with enriched data context. Before execution, it performs requirement understanding, schema comprehension, task decomposition, and step planning, then calls specific tools to gather evidence, finally synthesizing observations and the original request into a conclusion. Pre-planning constrains the search space; the Planner continuously incorporates expert knowledge and tool boundaries so each step maps to an executable capability.

Limitations emerged as capabilities expanded: (1) Planner system prompt grew unwieldy with web search, Yuque reading, Excel/Python, reporting, HTML/PPT/PDF, and push tools; (2) simple questions were forced through the full Plan-Execute-Solve flow, increasing interaction cost; (3) multi-turn context efficiency and accuracy suffered.

Phase 2: Dual Engine — Agentic Loop (ReAct) + Planning (ReWOO)

DataSage adopted a hybrid ReAct + ReWOO paradigm:

Agentic Loop (ReAct) serves as the main control flow: think → call tool → observe → rethink, dynamically deciding next steps based on task difficulty and real observations.

Planning Engine (ReWOO) handles complex task planning, generating structured analysis plans that constrain the overall search space when needed.

During execution, if unforeseen situations arise, ReAct adjusts based on actual observations.

This yielded DataSageCore, built on a minimal business-analysis state machine. Each model turn outputs a structured decision type: TOOL_CALL, DIRECT_ANSWER, COMPLETE, CLARIFICATION_REQUIRED, ERROR, INTERRUPTED. The system normalizes and post-checks decisions before invoking capabilities or entering terminal states. The state machine makes "whether execution conditions are met" a first-class judgment:

Missing required definitions → CLARIFICATION_REQUIRED (ask user, avoid tool guessing).

Context complete and question simple → DIRECT_ANSWER (skip full flow).

Need further exploration → TOOL_CALL (bring real observation back; Planner exposed as a special tool).

Task finished → COMPLETE (explicit convergence, not reliant on a tool's completion message).

The state machine handles orchestration only; domain analysis experience remains external. Complex planning is exposed as a meta-tool to DataSageCore, which passes user question, business context, data sources, and domain experience to the Planner, receiving a structured plan. Plan steps show what can proceed and what information must be filled. The plan supports HITL (Human-in-the-Loop): system presents preset steps, user adopts or modifies — e.g., changing time granularity, excluding order types, specifying core dimensions — which are treated as business fact supplementation, not ordinary feedback. DataSage also distinguishes the first decision from the AFTER_TOOL_RESULT phase; after entering the execution chain, the tool set narrows, avoiding repeated complex planning calls.

Standardized Pluggable Multi-Agent Tool System (Agent-as-Tool)

Pain Point: Tool Proliferation Degrades Top-Level Orchestration

With Agentic Loop as the main flow, tool calling becomes the core action. Data analysis capabilities are heterogeneous: internal RPC services, external agents, standard MCP (Model Context Protocol), standard Skills (reusable stable workflows). Examples: data retrieval via internal dataset/metric RPC; analysis via standard agent; Yuque doc read/write via MCP; risk business via Skill; reporting via internal agent. Flattening all into a single tool layer causes three problems: (1) tool set bloat — every tool's name, description, and I/O schema consumes context from turn one, increasing ambiguity and token cost; (2) execution detail leakage — query, calculation, and reporting internals flood the top-level DataSageCore; (3) scattered domain responsibilities — one DataSageCore must understand SQL generation/validation, statistical computation, Excel/Python, chart DSL, HTML, degrading orchestration quality.

Solution: Two-Layer Agent-as-Tool Structure

DataSage Harness adopts a Tool Calling + Multi-Agent System approach with Agent-as-Tool: the outer DataSageCore selects domain capabilities; each domain agent organizes its own Tools, Skills, MCPs, or internal DataSageCore instances within its responsibility boundary. The outer layer only needs to understand stable capabilities like query_agent, analysis_agent, skill_agent, mcp_agent, report_agent, not all low-level functions. Each domain agent has independent context and internal execution, completing multi-turn calls internally.

This split does not require uniform agent shapes. Fixed workflows stay code-controlled; open-ended tasks use internal AgentLoop; stable workflows use Skills; external systems connect via MCP. The key is shared stable boundaries for input, observation, context, and artifact.

Two Contracts and Standard Message Protocol DataSageAgent is the domain agent base class. DataSageCore reads AgentDefinition (metadata: code, description, parameters) and converts agent code, description, and parameters into tool definitions and JSON Schema. The two layers correspond to two contracts:

Outer Orchestration Contract: when this domain capability should be invoked, required business inputs, returned observation.

Inner Execution Contract: how this agent uses Tools, Skills, MCP, or code environments to complete the task.

Domain execution details are isolated by default, but the main flow may need key results. DataSage uses a standard message protocol to control cross-layer information flow: internal tool results are converted to generic Messages and appended to global Context; the Response returned to the outer layer is a structured summary containing task goal, key execution path, data result description, detail references, error reasons, and viable retry directions. The summary is generated by the producing domain agent because it best knows what the result means and what affects the next step. The outer DataSageCore neither re-interprets every low-level protocol nor gets flooded with full execution details.

Massive Data Context Management System

Pain Point: Data Semantic/Feature Loss in Context Passing

In the GMV example, the analysis continuously produces time-series, per-channel results, per-region results, anomaly samples, calculation processes, charts, user corrections, and intermediate hypotheses. Stuffing all into model context quickly hits token limits, causing token overflow , attention dilution in long contexts, and semantic drift . General agents typically use file summarization + head/tail truncation; for code, truncated context can be recovered via function/class/call relations and verified by compilation/tests. But when the artifact becomes a data table, keeping only headers and first few rows may drop tail anomalies; random row sampling cannot guarantee extremes, missing values, or rare groups appear. The question becomes: "Within a limited prompt budget, which data features must be preserved to support correct next-step analysis judgments?"

Solution: Dedicated Data Context Management

Layered Context and Deterministic Links

Referencing mature Agent Harness context designs, DataSage splits context into Raw History (append-only authoritative record of real messages, tool calls, and results) and Prompt View (budget-constrained content for the current model call: necessary task background, recent messages, processed data summaries). Data References point to accepted raw results, allowing domain agents or report flows to page-read details on demand. Compression only changes the next-turn Prompt View, never rewrites Raw History. Current problem, recent messages, and unclosed tool_call/tool_result relationships are protected; earlier closed plain text can be summarized or truncated; parsable data results enter a dedicated sampling summarization pipeline. This layering lets summarization serve planning and delegation while raw data retains factual responsibility.

Data Summarization: Statistical Profiling + Representative Row Sampling

DataSage's data summary has two parts:

Statistical Profiling (Statistical Portrait): computed on the full bounded result set — column types, distinct counts, missing rates, quantiles, mean, standard deviation, top frequent values.

Data Sampling (Representative Rows): retains real records and multi-column combinatorial relationships so the model sees concrete data shapes.

Before sampling, the system verifies the result can be reliably parsed as a regular table (currently fixed-column delimiter tables and JSON record arrays). It then identifies metric columns, date columns, high-cardinality identifiers, and ordinary dimensions using column names, numeric ratios, and value cardinality. High-cardinality identifiers (order IDs, user IDs) are excluded from stratification to prevent near-unique fields from exploding stratum counts. Default representative row budget is 20 rows, selected by priority:

Small tables: keep all rows.

Prioritize rows containing min/max of each metric column; if missing rate > 5%, keep at least one null record.

Ordinary dimensions stratified by single or multi-column combinations; low-frequency strata get representative rows first, then common strata supplemented by stratum size.

Remaining slots filled with fixed random seed for reproducibility.

Statistical profiles always come from the full table, never re-estimated from the 20 representative rows. The summary header explicitly marks original table size, sampling method, and "this is not full data" to prevent the model from treating the sample as the population. When column count is high or rows are long, the system preferentially reduces representative rows while preserving schema and statistical profiles.

Current production path uses a reproducible, cost-controlled compromise. Bucket-by-date, recent-biased, standardized farthest-point sampling, Neyman allocation (stratified sample size allocation by stratum size), and reservoir sampling (equal-probability sampling for unknown-size streams) each suit specific tasks or remain under research; they are not unified into the main path. Sampling strategy is driven by use case, not algorithm count.

Sampling Quality Fence: Compression Pass ≠ Analysis Correct

Online self-checks focus on two most-loss-prone information types: whether metric extremes appear in representative rows, and how many low-frequency categories (≤5% frequency) are retained. Extreme retention accounts for 60% of quality score, rare category recall 40%; composite score ≥ 0.7 passes. On failure, the summary shows a fidelity warning but does not block the main flow because full data remains accessible via references. Model understanding consistency between full data and summary is further evaluated on fixed task sets. The boundary is clear: context summarization aids agent planning, delegation, and next-step selection; it cannot bear final factual responsibility in reports. Numerical values, anomaly explanations, and attribution conclusions must still revert to accepted raw data and deterministic computation results.

Traceable Calculation Definitions and Analysis Evidence Chains

Pain Point: Tool Success ≠ Analysis Delivery Success

From the user question: "Last week's GMV main fluctuation came from XXX channel." Trust depends on traceability to explicit GMV definition, accepted data, and verifiable contribution calculation. Data analysis lacks a unified completion signal:

SQL success only proves query execution, not metric correctness.

Python normal exit doesn't prove input range and calculation rules.

Chart renders doesn't prove chart-data consistency.

Report structure complete doesn't prove every conclusion has upstream evidence.

Push API success doesn't prove user received final version.

Therefore, model output "analysis complete" or tool success is insufficient as DataSage Harness terminal state.

Solution: Four-Layer Completion Decomposition

DataSage decomposes "complete" into four layers, each must hold sequentially:

Action Complete: query, calculation, file generation, or model call finishes normally.

Artifact Trusted: result satisfies structural constraints, accepted, persisted, and re-readable.

Conclusion Evidenced: report facts, figures, and judgments trace back to accepted data and derivation process.

Delivery Closed: body, charts, files, and user-facing card originate from the same result; system confirms full delivery.

These layers shift validation from "did a tool succeed?" to "is the entire analysis chain trustworthy?"

Stage 1: Verifiable Calculation Definitions, Traceable Data Results

A flowable analysis result requires metric definitions, raw data details, result summaries, and visualizations to correspond. Upon receiving an artifact, the system checks structure, scale, data types, visualization content against constraints, and artifact-task affiliation. Passed results are persisted and re-read to confirm write-restore integrity. Subsequent flows access data only via controlled references, re-verifying task identity and data consistency on read. Failed validation revokes the new reference; failed results never mix into accepted artifacts. At this point the system can answer "does this analysis result truly exist and belong to the current task?" — giving the report flow reliable inputs.

Stage 2: Evidence Pool and Logic-Driven Report Generation

The report agent collects analysis steps and valid artifacts before writing. Ordinary queries, Excel/lightweight analyses, and domain agent final results enter a unified evidence pool; failed, discarded, or report-type results are excluded; duplicates merged. The report agent uses only filtered evidence references and execution traces. When details are needed, it re-reads corresponding data via a whitelist, not guessing from the full conversation. With the evidence pool ready, the report outline constrains each chapter: what question it answers, which evidence is allowed, which charts may be shown. Key judgments in the body must bind data references, source facts, and necessary derivations. Even if the user only wants the conclusion, the system first establishes the conclusion-source relationship.

Having references ≠ correct references. The report agent runs two checks:

Deterministic Rule Checks: required question coverage, reference overreach, fact omission, numeric drift, chart binding.

Semantic Review: whether the report answers the user question, argument coherence, constraint disclosure.

On issues, the system repairs affected content per issue type, rebuilding drafts if necessary. Passed body converges to a structured report, then generates Markdown and HTML. Presentation layer may adjust layout and style but cannot recreate numbers, conclusions, or data relationships.

Stage 3: Reflective Audit with Calculation Definition Verification and Evidence Traceability

The report agent's further direction: explicitly assign fact authority, computation authority, reference authority, and completion state to system management. The model continues to understand the problem, organize arguments, write within constrained evidence, detect semantic issues, and propose fixes; the system handles raw source persistence, fact-chapter attribution, deterministic computation, evidence verification, repair scope, body freezing, and delivery confirmation. Key designs include:

Freeze available data sources, metric definitions, units, time ranges, dimensions, filter scopes, trust levels, and constraints before report start.

Ratios, aggregations, sorting, unit conversions, and derived metrics computed deterministically with recorded input facts, operation rules, and output facts.

Each chapter receives only its responsible question, allowed evidence, and confirmed upstream conclusions.

Every fact-bearing visible text must bind specific facts, conclusions, and evidence references.

System determines repair scope by issue ownership and chapter dependencies; affected chapters regenerate and submit proof; others stay unchanged.

After all quality checks pass, the body is frozen as the single content source; Markdown, HTML, and result metadata are generated from it. Registration, archiving, and UI card sending complete, then the system produces a delivery receipt. Only when infrastructure confirms full delivery can the outer task enter success state; unconfirmable chains remain in shadow validation and cannot turn "report generated" into "report successfully delivered."

Summary: Evidence Chain Guarantees Data Trustworthiness

Before the weekly GMV report delivers, it must satisfy: (1) report GMV definition matches extraction definition; (2) anomaly judgment uses explicit baseline and deterministic calculation; (3) dimension attribution traces to corresponding data and contribution calculation; (4) charts, tables, and body reference the same fact set; (5) DingTalk content comes from the checked-and-frozen final version. DataSage ensures trustworthiness not by model "sounding reasonable" but by system-preserved relationships among definitions, data, derivations, presentations, and delivery.

Overall Summary: Four Breakpoints, One Division of Labor

Returning to the GMV question, it tests not whether the model can write SQL or a report, but whether the system can turn a business question into a verifiable delivery chain. DataSage Harness builds along the chain's four breakpoints:

Requirement understanding & orchestration drift → Dual Engine: Agentic Loop decides dynamically on real observations; Planning Engine distills domain experience and constrains search space; decision state machine makes "execution readiness" a first-class judgment, turning vague queries into confirmable, executable plans.

Tool expansion inefficiency → Multi-Agent + Agent-as-Tool two-layer structure: Outer DataSageCore orchestrates only a few stable domain agents; domain agents organize Tools, Skills, MCP internally. Capability growth no longer costs top-level context bloat and orchestration degradation.

Long/multi-turn instability → Data Context Management: Raw History / Prompt View layering; statistical profiling + representative row sampling preserves core data features; quality fence verifies fidelity; raw facts always re-readable via references.

Opaque definitions & lineage → Traceable Evidence Chain: "Complete" split into action done, artifact trusted, conclusion evidenced, delivery closed — layered acceptance; every report number traces back to definition, data, and derivation.

Running through all four solutions is the same division of labor: models handle understanding, hypothesizing, path organization, and interpretation; systems handle definition constraints, state management, deterministic computation, evidence verification, and delivery confirmation. Thus, the GMV task's completion state is not model generation stop nor a tool's success return, but confirmed definitions, traceable process, verifiable conclusions, and delivered report. This is DataSage Harness's positioning: making model capabilities a verifiable data analysis production process in the business semantic environment.

Next Phase: Self-Evolving Data Analysis Agents

The four solutions focus on single-task quality and context accumulation. The next question: can DataSage turn each analysis into reusable experience so the system grows stronger instead of restarting from the same baseline? Core elements already exist in the solving process: user-confirmed/modified analysis plans, repeatedly used stable analysis flows, cross-task accumulated definitions and preferences, report review findings, and post-delivery follow-ups. The key is the distillation method — a single task's success ≠ reusable experience; a successful attribution path may be a coincidence under a specific data distribution. DataSage must precipitate these trajectories with provenance, scope, and timeliness into retrievable assets: analysis SOPs, Skills, definition confirmation templates, and Planner domain decomposition experience; injected on-demand at new task start, with usage effects feeding back to decide asset retention, update, or retirement. Experience extraction and activation must be verifiable and traceable; fact authority and acceptance authority remain with the system and humans, otherwise erroneous experiences self-reinforce, becoming more dangerous than no experience. When every task's understanding, execution, context, and evidence are fully recorded, the system gains the foundation to turn "what was done" into "next time's capability."