From Labels to TaskSpecs: How Production Agents Structure Intent Recognition

1. Problem Analysis

Agent intent recognition is often reduced to a simple classifier: knowledge questions go to RAG, data queries to SQL, chitchat to the LLM. This works for basic Q&A bots but fails when agents handle multi-step tasks. A single label cannot capture a request like "Check why refunds rose in East China yesterday, summarize, and send to the ops group" — which contains at least four actions (query, analyze, compose, send), key parameters (yesterday, East China, refund), and a final step with real external side-effects and different risk than read-only operations. If the system only outputs a data_analysis label, downstream execution still lacks ordering, missing information, and confirmation points.

Interviewers asking about intent recognition care less about which model classifies and more about how natural language becomes a verifiable, routable, traceable task description. Intent recognition is the agent's control entry point: get it right and the planner and tool calls have reliable input; get it wrong and even the strongest models and tools just execute errors more efficiently.

1.1 Define the Recognition Output First

Engineering starts not by picking a model but by defining what "recognition complete" must produce. An executable recognition result is organized as a structured TaskSpec with core fields: dialogue act, business domain, actions, entities and slots, subtasks, risk level, missing information, candidate routes, and version. For the refund example a simplified result looks like:

{
  "dialogue_act": "request",
  "domain": "refund_operations",
  "subtasks": ["query", "analyze", "compose", "send"],
  "slots": {"date": "昨天", "region": "华东区", "metric": "退款"},
  "missing_slots": ["target_group"],
  "risk": "external_write",
  "route": "operations_agent"
}

Two boundaries matter. First, intent ≠ tool. query means the user wants data; whether SQL, a metrics platform, or RAG is used is decided later by the tool router based on data source, permissions, and availability. Adding a new intent per tool makes the label system unmanageable. Second, intent recognition does not produce the full execution plan; it provides a reliable task contract. Complex decomposition, parallelism, and retries belong to the Planner. This separation lets recognition, planning, and execution be evaluated independently and simplifies debugging.

The intent taxonomy uses a layered design. Layer one: stable coarse categories (knowledge query, data operation, content generation, external action, conversation control). Layer two: concrete business actions. Dialogue acts like confirm, cancel, correct, supplement are modeled separately because "Yes, send it" is unintelligible without the pending confirmation state. The taxonomy must retain unknown or out_of_domain — better to admit failure than force every input into the nearest known label.

1.2 Restore the Current Utterance Fully

Hard production inputs are rarely complete sentences; they are follow-ups like "Still use yesterday's definition", "Just send it to him", "Don't send". Classifying only the last utterance is meaningless. Before classification, context parsing must read recent turns, current task state, pending confirmations, filled slots, and user permissions to reconstruct a self-contained request while preserving the raw input for audit.

This step must not dump unlimited history. The system takes only task-relevant state and distinguishes facts, model inferences, and explicit user confirmations. If "he" could refer to two people, the model must not silently pick one; instead target_user is marked as a missing slot and the user is asked. When the user says "Don't send", the conversation-control intent overrides the original send plan and immediately cancels the pending action.

1.3 Hierarchical Routing Controls Cost and Misclassification

Sending every request to a large LLM for free-form judgment is fast to develop but poor on latency, cost, and stability. Production uses cascaded routing: deterministic requests take a short path; only hard cases escalate.

Layer 1 – Deterministic gates : handle cancel, confirm, permission checks, fixed commands, and explicit safety boundaries. Rules are clear and misclassification cost is high; probability models should not be the sole decider.

Layer 2 – Lightweight classifier : covers stable, high-frequency closed-set intents. Few intents → direct classification; many intents → embedding-based retrieval of Top-K candidates from intent descriptions and examples, then a small model discriminates among similar candidates. Key: provide not only positive examples but also explicit boundaries and negative examples for confusable pairs (e.g., "query refund policy" = knowledge query vs. "execute refund" = fund operation).

Layer 3 – LLM semantic routing : handles complex phrasing, multi-intent, close candidates, or novel requests. Input includes only retrieved candidate intents, necessary context, and a strict JSON Schema; output must be candidate ranking, slots, and missing items — no free-form generation. Schema validation failure allows limited retries, not infinite loops. If still undecided, escalate to clarification or safe fallback, never force a label.

1.4 Decompose Multi-Intent but Don't Execute at Recognition Layer

Real requests often contain multiple actions. The right approach is not to pick a "primary intent" and discard the rest, but to output ordered, dependent subtasks. In "query cause, summarize, send group", query must finish first, analysis depends on query results, composition depends on analysis, and send depends on target group and user confirmation. The planner receives this small task graph and decides parallelism, slot-filling pauses, and mandatory stops.

This decoupling lets the recognition result pass through a policy layer first. Read-only steps can tolerate slight routing drift and fall back; send-message, place-order, refund, delete-data actions must pass permission checks, parameter preview, and explicit confirmation. The model identifying "user wants to do X" does not grant "system may execute X now" authorization.

1.5 Confidence Is Not a Self-Reported Number

Having the LLM emit confidence: 0.93 in JSON is convenient but uncalibrated and unsafe for production control. Production systems fuse multiple signals: closed-set classifier probability and Top1-Top2 margin, semantic distance to known intent examples, cross-router agreement, required-slot completeness, out-of-distribution detection, and rule-validation conflicts. Thresholds are then calibrated on a labeled validation set, observing misrouting and rejection costs at each threshold.

Thresholds must vary with risk. High-confidence knowledge queries can execute directly; medium-confidence low-risk requests can upgrade to a stronger model; external writes (even with clear intent) require action preview and confirmation; low-confidence or OOD requests trigger the highest-information-gain clarification question — not "please be clearer" but "Do you want to query the refund policy or execute a refund?" — one question that splits the most confusable boundary.

1.6 Common Pitfalls

Most traps appear during label design. Too coarse: different workflows crammed into one label, making the recognition result non-executable. Too fine: two labels that even annotators confuse, so the model cannot be stable. For chronically confused labels, first verify they truly represent different user goals. If the difference is only a parameter, merge into one intent and express via slots instead of proliferating labels.

Another frequent mistake: stuffing classification, planning, and tool calling into one prompt. Demos look short, but production errors become impossible to attribute — was it intent, missing parameter, or wrong tool? Some teams only re-run standard test sets on model upgrades, lacking user corrections, topic switches, and adjacent-intent negative examples. Overall scores rise while business-critical boundary errors increase. The goal of intent recognition is not "model always gives an answer" but "system knows when to act and when to stop and confirm".

1.7 Evaluation Must Track End-to-End Routing Correctness

Offline evaluation cannot rely on a single overall Accuracy. Intent distribution is highly imbalanced; common intents mask long-tail errors. At minimum track Macro-F1, key-intent recall, confusion matrix, OOD rejection rate, slot F1, and joint intent+slot correctness. For money, delete, and external-send intents, separately measure false-trigger rate because one wrong execution costs far more than an extra clarification.

Test sets must cover adjacent-intent hard cases, multi-turn ellipsis and coreference, single requests with multiple actions, mid-stream corrections, privilege-escalation attempts, and completely unseen intents. Split data by user or task to avoid the same phrasing appearing in both train and test, which inflates scores.

Online metrics that matter: misrouting rate, post-clarification task success rate, user correction/retraction rate, tool-call success rate, end-to-end latency, and per-request cost. A full trace records raw input, context parsing, candidate intents, final routing, and tool results (with PII redacted). User corrections and failed traces become new hard cases. Every label, prompt, model, or threshold change runs the same regression suite and is first observed via shadow traffic to prevent local accuracy gains from increasing high-risk false triggers.

Returning to the opening example: the system first resolves "yesterday" to an actual date and "East China" to the business definition, then identifies four dependent subtasks (query, analyze, compose, send). The first three run if permissions allow; after generating a preview, the system asks for the target group and explicit confirmation because the last step is an external write. If the user then says "Only analyze, don't send", the conversation-control layer cancels the send node instead of continuing the old plan. This complete chain is what makes intent recognition truly usable in production.

2. Reference Answer

We did not build intent recognition as a single LLM classification. Instead, it serves as the agent's control entry point. Incoming requests first undergo context completion using recent dialogue turns, current task state, and filled slots to resolve ellipsis and coreference. Then a layered routing handles the request: cancel, confirm, permission, and high-risk boundaries are processed by rules; high-frequency closed-set intents use a lightweight classifier; when many labels exist, embedding-based retrieval fetches Top-K candidates; complex, multi-intent, or novel phrasing escalates to an LLM with a fixed JSON Schema outputting domain, actions, slots, missing items, subtasks, and risk. The recognition layer only produces a TaskSpec; the Planner decomposes tasks, and the Tool Router selects concrete tools.

For low confidence we do not trust the model's self-reported score. Instead we fuse Top1-Top2 margin, semantic distance, slot completeness, router agreement, and OOD signals, with thresholds set per risk level. Read-only queries may proceed or upgrade models; external writes (send, refund, delete) require preview and confirmation even at high confidence. Offline we track Macro-F1, key-intent recall, OOD rejection, slot joint accuracy, and high-risk false-trigger rates. Online we monitor misrouting, post-clarification success, user correction rate, tool-call success, latency, and cost. Real misjudgments are continuously added to regression suites, and every change is validated via shadow traffic to avoid increasing high-risk false triggers while improving local accuracy. The core principle: the model understands, the policy layer decides execution.