Decision Models Cut AI Inference Costs 98%, But 97% Accuracy ≠ 63% Document Success

Google Gemini Decision 1: Separating Judgment from Generation

On November 11, Google released Gemini Decision 1 (gemini-decision-1) via a new Decisions API . Unlike generative LLMs that produce text token-by-token, this API accepts multimodal input (text, image, audio, video, PDF up to 256K tokens) alongside a list of up to 1,000 independent questions with predefined candidate answers. The model returns structured probabilities for each candidate in a single forward pass, eliminating the need for repeated full-context generation.

The API supports three judgment types:

Noul (Yes/No) – e.g., "Is a required contract attachment missing?"

Choice (Multi-class) – e.g., "Which processing queue should this exception order enter?"

Score (Grading) – e.g., "Assign risk priority level."

Google reports median model compute latency of 50–100 ms and end-to-end latency of 150–250 ms for contexts ≤16K tokens with 1–100 questions. Pricing is $0.10 per million input tokens; output tokens are free. Shared input is billed once, though each question's instructions add to token count. In shared benchmarks, Gemini Decision 1 costs nearly 5× less than calling Gemini 3.5 Flash-Lite via the Interactions API.

Decision models are not drop-in replacements for open-ended generation (e.g., clause extraction, dispute explanation). They excel at choosing among defined options under explicit criteria.

Amazon Case Study: 98% Inference Cost Reduction with Strands Decider E4B

On November 9, the Strands Agents team disclosed an internal Amazon migration: replacing Claude Sonnet 4.6 with Strands Decider E4B for classifying 20 categories of internal tickets (upgrade requests, resource requests, alerts, security events). The task only required picking a single label from a fixed set.

Key engineering changes:

Compressed lengthy category descriptions into 1–2 sentence decision rules.

Added a "None applicable" candidate.

Trimmed early comments on long tickets to fit context limits.

Ran A/B tests: started with Strands Decider 2B, moved to E4B, tuned confidence threshold for auto-classification.

Results (per million tickets):

Average latency: 4,215 ms → 897 ms ( −78% )

Input tokens: 6,554 → 2,009 ( −69% )

Output tokens: 170 → 1 ( −99% )

Total tokens: 6,724 → 2,010 ( −70% )

Inference cost: ~$22,200 → ~$500 ( −98% , ~1/44th)

Caveats: The 98% figure reflects model inference cost only, not end-to-end ticket processing cost. Billing models differ (Bedrock per-token vs. SageMaker instance-hour). The team used a three-stage rollout: old model only → parallel run (old model decides, new model logs discrepancies) → new model only, enabling safe fallback.

Prerequisites for such replacement: stable candidate set, closed-choice task, controllable input length, sufficient volume. If categories change frequently, detailed explanations are needed, or massive history must be analyzed, generative models may remain better.

Benchmark Gap: 97% Per-Item Accuracy Yields Only 63% Whole-Document Pass Rate

Artificial Analysis's Decision Index v2 (Nov 8) introduced AA-Paperclip : 1,500 documents (invoices, receipts, résumés, AI papers), each requiring 15 tasks (10 fact extractions + 5 judgments). A document passes only if all 15 are correct.

Results for Drex Max v1.1 vs. Claude Haiku 5.5 (Max) :

Per-item accuracy: 97% vs. 99%

Full-document pass rate: 63% vs. 92%

Under independence assumption, 0.97¹⁵ ≈ 63.3%, matching Drex's observed 63%. Claude's 99% per-item would predict 86% pass, but actual is 92%, showing errors are not independent and task difficulty varies.

Breakdown for Drex Max v1.1:

All 10 extractions correct: 88% of documents

All 5 judgments correct: 71% of documents

This reveals extraction and judgment are distinct capabilities. A model can extract every field perfectly yet still misjudge the business decision (e.g., invoice amount correct but payment condition mis-evaluated). Conversely, a correct judgment doesn't prove all intermediate fields were right.

Real-world business errors have unequal severity. Treating a missing non-critical remark the same as an unauthorized payee change leads to poor automation thresholds. Full-task pass rate measures simultaneous correctness; business acceptance must also define which errors are recoverable vs. must-block.

Confidence Scores ≠ Reliability: Calibration and Consistency Matter

Decision models output candidate probabilities, enabling routing: high confidence → auto-process; low confidence → human/stronger model. But probability calibration is essential: a reported 0.95 confidence must correspond to ~95% empirical accuracy on representative historical data.

Artificial Analysis now includes calibration in its composite score. AA-Symmetry tests whether equivalent questions (rephrased, candidate order shuffled, labels swapped) yield consistent judgments. Amazon found that merely reordering JSON keys changed output probabilities by up to 0.19; they fixed this by canonicalizing key order.

Reliability requires three pillars: correctness, calibration, and input-perturbation consistency. Moreover, model probabilities cannot override deterministic business rules (approved vendor list, authorization limits, duplicate-payment checks). A sound architecture separates concerns:

Extraction models → fact retrieval

Decision models → candidate judgment

Rule systems → hard constraints

Workflow engines → execution & audit

Human review → exceptions & accountability, with feedback loops

In finance, procurement, supply chain, decisions depend on live business state, permissions, approval records, contracts — not just input text.

The Real Metric: Cost Per Correctly Completed Business Transaction

Lower per-call inference cost doesn't guarantee lower per-successful-business cost. A simplified illustration (not real data):

Scheme A: model cost ¥0.10, 20% human review (¥8 each) → avg ¥1.70

Scheme B: model cost ¥0.60, 5% human review (¥8 each) → avg ¥1.00

Scheme A's model is 6× cheaper, but higher fallback rate makes it more expensive overall. Token price alone is misleading; input reuse matters. In AA-Paperclip, GLiDE v1.2 re-processes the document per question (~63K tokens/task, ~$780/M tasks) while Drex Max v1.1 shares input (~9K tokens/task, ~$102/M tasks).

Enterprises should track three linked metrics:

Whole-document first-pass rate – % of transactions meeting acceptance criteria without rework.

False-pass rate & auto-coverage – % of should-block transactions incorrectly auto-approved, and % of transactions auto-completed within risk limits. Must be viewed together; raising threshold to improve accuracy reduces coverage.

Cost per correctly completed transaction – (model inference + data processing + human review + retries + infrastructure) ÷ (transactions passing acceptance). High-impact low-frequency errors need separate risk-loss accounting.

Practical pilot: pick a high-frequency, well-bounded decision task. Build a test set from historical data covering normal, edge, and high-risk cases. Run old and new pipelines side-by-side, capturing judgments, confidence, human routing, final outcomes. Sweep automation thresholds, observe false-pass rate, human load, total cost. Only when the new scheme stably lowers unit completion cost under fixed risk constraints does inference savings translate to business value.

The deeper shift: AI cost optimization is moving from model internals (architecture, deployment, API design) to business processes (task decomposition, error economics, human–machine division of labor). Google's decision API, Amazon's production swap, and third-party benchmarks all point to the same conclusion: as agents take on real production judgments, the unit of AI value measurement must shift from one model call to one correctly completed business transaction .