Designing a Production-Ready Business Analysis Skill for Enterprise AI

Overview

The author extends the previous discussion on trustworthy data‑query skills to the full "analyze this month’s business performance" scenario. The key insight is that business‑analysis is not a single question but a chain of dependent tasks that must be orchestrated deterministically.

Production Architecture of the Business‑Analysis Skill

A Skill is a predefined multi‑step execution plan that specifies trigger conditions, required parameters, execution order, tool calls, validation points, stop/reject criteria, and final output structure. The Agent orchestrates the flow while the LLM only handles language‑intensive steps.

Report Scope Confirmation → Modular Data Retrieval → Deterministic Computation → Anomaly Detection & Variance Decomposition → Business Evidence Linking → Insight & Chart Generation → Independent Review → Human Confirmation & Versioning

The architecture diagram (see image) visualises this pipeline.

1. Modularization: Do Not Generate the Whole Report at Once

Instead of feeding all data into a single LLM prompt, the report is split into independent modules (Revenue, Cost, Expense, Profit, Summary). Each module follows the same execution pattern:

Module Start → Load Confirmed Period/Scope → Call Pre‑defined Data Retrieval Tool → Perform Deterministic Computation → Identify Anomalies & Contributions → Generate Tables & Charts → LLM Generates Text Insight → Independent Review → Save Module Version

Benefits:

Error localization – a failure in the Revenue module does not require re‑generating the entire report.

Auditability – finance staff can review and adjust each module separately.

Checkpoint recovery – time‑outs or model failures can resume from the last successful module.

Modules must respect explicit dependencies (e.g., the Profit module consumes verified Revenue and Cost results; the Summary module only reads audited module outputs).

2. Tool Design – Deterministic Tasks Belong to Code

Each step that can be expressed as a contract is implemented as a Tool with a clear responsibility and deterministic implementation: query_income_data: retrieve income by product, customer, organization (predefined SQL template). query_budget_data: fetch budget and version information (deterministic query). calculate_metrics: compute MTD/QTD/YTD, YoY, completion rates, structural ratios (pure code). detect_variance: flag anomalies using amount, proportion, and business importance (rule‑based). decompose_variance: break price, volume, and interaction effects (deterministic algorithm). retrieve_business_events: pull activity, price‑change, customer, and organization events from governed sources. generate_chart: render bar, trend, and waterfall charts (template rendering). generate_insight: translate structured results into natural language (LLM).

Every Tool must return a structured result, data version, definition, and execution status; failures must be explicit so the Agent cannot guess success from free‑form logs.

Decision rule: If repeated identical inputs must yield identical outputs, the task belongs to code/rules/templates, not to the LLM.

3. Attribution Analysis – Compute Contribution Before Explaining Cause

Two concepts are distinguished:

Variance Attribution : mathematically decompose total change into product, customer, region, price, volume, etc., answering "where did the change come from?"

Causal Judgment : prove that a specific action caused the change, answering "why did it happen?" (requires experiments or business evidence).

Attribution flow:

Detect Anomaly → Define Benchmark → Choose Decomposition Model → Drill‑down by Business Dimensions → Compute Contributions → Link Business Events → Output Attribution, Confidence, and Unexplained Items

Key rules:

Anomaly detection must consider absolute amount, proportion, business importance, and small‑base effects.

Budget, YoY, and MoM differences are separate benchmarks and must not be mixed.

Decomposition methods differ by metric type (additive, multiplicative, ratio, multi‑factor) and must be predefined.

When total change is near zero, use absolute contribution or basis‑point change instead of percentage contribution to avoid distortion.

Controlled drill‑down respects thresholds for minimum amount, minimum contribution share, maximum depth, and candidate count.

All sub‑contributions must round‑trip to the total within tolerance; otherwise the system aborts deterministic attribution.

Evidence grading distinguishes Data Phenomenon (directly observable), Analytical Explanation (supported by decomposition), and Hypothesis (requires further validation).

4. Insight Generation – Only Explain Verified Results

Four hard constraints ensure trustworthy insights:

Controlled input – LLM receives only validated, structured results; it cannot perform additional queries or calculations.

Evidence‑backed conclusions – every core finding must cite specific metrics, values, and benchmarks.

Fact layering – output separates data phenomenon, analytical explanation, and hypothesis.

Fixed format – the report follows a predefined structure: Business Overview, Core Findings, Major Variances & Attribution, Risk Alerts, Recommendations & Validation Items.

This format enables month‑over‑month comparison and systematic audit.

5. Review – Challenge Before Delivery

After generation, an independent reviewer performs a checklist covering consistency of periods, data versions, back‑tracking of attribution, evidence for causal statements, avoidance of causal hallucination, and avoidance of un‑reviewed data in the Summary.

Lightweight checks suffice for simple queries; full‑scale reports require a robust review chain involving deterministic checks, reviewer validation, and final business sign‑off.

6. State & Version – Recover, Modify, and Trace

Production systems must support:

Checkpoint saving – record inputs, query results, snapshots, calculations, charts, model I/O, and review outcomes for each module.

Partial re‑run – only re‑execute modules affected by data updates or wording changes.

Manual revision – allow finance or business owners to edit conclusions while preserving the original Agent output, diff, and rationale.

Traceable versions – each module keeps Agent, Review, and Human‑confirmed versions with timestamps, authors, data versions, metric versions, and Skill version.

The final report is assembled from audited modules rather than a monolithic text.

7. Evaluation – Beyond Surface Quality

Key metrics for continuous evaluation include:

Data‑retrieval accuracy (>90% core chain match).

Time‑window accuracy (MTD/QTD/YTD dates & aggregation 100%).

Attribution back‑track rate (100% of variance explained).

Evidence compliance rate (every "reason" has traceable evidence, 100%).

Module completion rate (successful generation & saving within SLA).

Manual edit rate (track substantive business edits vs cosmetic changes).

P95 latency (e.g., <5 minutes from trigger to first draft).

Test sets must cover normal months and edge cases such as missing data, incomplete close‑out, near‑zero changes, small‑base volatility, budget version switches, organizational changes, metric definition updates, un‑explainable attribution, and conflicting business events.

8. Common Pitfalls & Mitigations

Skipping data foundation and feeding raw model – fix by building core data assets first.

Treating the Agent as a universal entry point – implement intent and complexity routing.

Letting LLM compute MTD/QTD/YTD on the fly – use deterministic time services.

Allowing LLM to query during insight generation – pre‑compute and validate all inputs.

Generating the whole report in one pass – adopt modularization, checkpoints, and partial recovery.

Writing causal statements without evidence – enforce variance attribution, evidence grading, and hypothesis labeling.

Identifying anomalies solely by rate – combine amount, rate, and business thresholds.

Attribution that cannot back‑track – retain residual items and perform tolerance checks.

Neglecting Skill and metric documentation – enforce same‑repo PR gating and regression tests.

Evaluating only textual quality – layer evaluation of data, computation, and evidence correctness.

Conclusion

The Business‑Analysis Skill transforms existing organizational analysis methods into an executable, auditable, and maintainable production line. Deterministic data retrieval, code‑driven calculations, and clear tool contracts keep the pipeline stable; the Agent orchestrates modules; the LLM merely translates verified evidence into managerial language. Independent review, version control, and rigorous evaluation ensure that the system remains trustworthy from demo to production.