Building a Trustworthy Data Query Skill for Enterprise AI: Architecture and Best Practices

Why "Chat + Warehouse" Feels Fluent but Lacks Trust

Although large models can generate code, data analysis requires a single, trustworthy source and precise definitions; the real challenge is mapping natural‑language questions to the correct data source, fields, and calculation logic.

Three common failure modes identified by Company A:

Concept‑field mismatch: Questions like “active users last month” have hundreds of candidate fields; the system must clarify the exact business definition.

Context expiration: Data sources, business definitions, and schemas evolve, causing answers that were correct at launch to drift from reality (offline accuracy drops from ~95% to ~65% after a month without Skill maintenance).

Information not found: Even when historical notebooks and SQL exist, the agent often cannot locate the relevant answer (≈80% of correct answers were hidden in existing artifacts).

Skill as a Behavioral Contract

A well‑designed Skill does not merely dump schema and rules into a long document; it organizes six instruction types that govern agent behavior:

Trigger condition: Determines which questions enter the Skill.

Source priority: Enforces semantic‑layer precedence over raw SQL.

Clarification & refusal: Requires confirmation when definitions are unclear and stops execution for out‑of‑scope requests.

Execution order: Encodes the steps of a senior analyst into a repeatable workflow.

Validation rules: Mandates independent audit of queries; the generator cannot self‑validate.

Output contract: Forces the answer to include source, freshness, owner, constraints, and trust level.

Skill only dictates behavior; actual permission checks, SQL safety, and resource limits are enforced by external systems.

Intent Router: First Decision Point

The system first decides whether it can handle a request. Routing strategies include:

Rule + keyword matching for high‑frequency, well‑bounded scenarios.

LLM‑based classification with prompts and examples for long‑tail expressions.

Hybrid approach: rules take precedence, falling back to LLM with low confidence prompting for clarification.

Requests that fall outside the Skill’s responsibility are explicitly refused.

Parameter Completion: Compiling Business Parameters

After routing, the system clarifies period, accumulation method, metric definition, scope, and currency. Examples:

Period: Use a unified financial calendar to resolve “last month”.

Accumulation: Convert MTD/QTD/YTD into explicit start‑end dates.

Metric definition: Confirm allocation method (full, half, none).

Scope: Map to group, business unit, or product line based on user role.

Currency: Default to reporting currency but annotate the answer.

Time expressions follow strict rules (e.g., “last week” means the previous full week, not the past 7 days) and use MAX(date) to assess freshness.

Text‑to‑Metric vs. Text‑to‑SQL

Standardized queries first hit the governed semantic layer (Text‑to‑Metric). Only when the metric layer cannot satisfy the request does the system fall back to Text‑to‑SQL for long‑tail cases. The fallback process includes:

Retrieve only relevant schema instead of injecting the whole catalog.

Explicitly inject field descriptions (name, meaning, enum, unit, pitfalls).

Provide few‑shot examples covering dimensions, time ranges, filters, and joins.

Output a query plan (metrics, dimensions, filters, joins) before generating SQL for verification.

Two Skill types support this:

Knowledge Skill: A navigation map that tells the agent where to look.

Process Skill: An operation manual that constrains how to act once the data is located.

If both metric and SQL paths fail, the system returns a clear “cannot answer now” message with the failure reason and queues the question for future metric creation.

Reviewer: Post‑Query Audit

Every result passes an independent reviewer that checks data source selection, metric applicability, time‑zone correctness, denominator and granularity, join completeness, bias, and whether conclusions exceed data support. This audit improves accuracy by ~6 % while adding ~32 % token cost and ~72 % latency; the cost can be limited to high‑risk queries.

Output Contract

Answers must include a footer with data‑source hierarchy, trust level, audit status, freshness, and owner. For power users, the UI should allow expanding to view metric definitions, source details, and raw SQL.

Permission & Auditing

Permission checks are enforced at the data‑layer: the permission service returns accessible tables/fields, and the database applies row‑level security or secure views. SQL statements undergo AST‑level validation (read‑only, no SELECT *, mandatory LIMIT, resource quotas). Every interaction is logged with user, timestamp, original question, intent, metric hit, generated SQL, validation outcome, response, and latency. Logs support compliance and continuous test‑set generation.

Knowledge Maintenance

Skills should contain stable IDs, business definitions, applicable scope, formulas, aggregation types, dimensions, time‑window support, trusted data sources, lineage, refresh frequency, owners, version, verification timestamps, deprecation status, and typical questions. Updates are triggered by data‑model changes, business‑rule revisions, online error corrections, and periodic inspections. Company A stores Skills alongside data‑model code; ~90 % of data‑model PRs include corresponding Skill changes, and CI checks enforce cross‑layer consistency.

Evaluation

Three‑layer testing ensures robustness:

L1 – Basic capability: Public or synthetic data to gauge model limits.

L2 – Business scenarios: 50‑100 real business questions to measure accuracy on actual schemas and definitions.

L3 – Question consistency: 3‑5 paraphrases of the same question to verify answer stability.

Post‑launch monitoring tracks metric hit rate, retrieval accuracy, time‑window correctness, question‑consistency, refusal reasons, user‑error types, and P95 latency. Model upgrades, knowledge‑base updates, metric changes, or schema migrations trigger regression suites; audit logs capture the exact Skill, metric, and document versions used for each answer.

Conclusion

The trustworthy data‑query pipeline consists of Intent Router → Parameter Standardization → Metric‑first retrieval → SQL fallback → Reviewer → Output contract, all guarded by permission enforcement, knowledge maintenance, and continuous evaluation. A well‑crafted Skill is not a lengthy manual but an executable behavior contract that ensures clear rules, limited entry points, early error blocking, and verifiable changes, marking the transition from demo to production for enterprise AI agents.