Skip to content
Decision Models Near GPT-6: 3-Point Gap, 40x Faster, 15x Cheaper

Decision Models Near GPT-6: 3-Point Gap, 40x Faster, 15x Cheaper

Artificial Analysis's Decision Index v2 shows specialized decision models like Drex Max v1.1 scoring 73 versus GPT-6 Luna's 76, but with 40x lower latency and 15x lower cost per million tasks, suggesting enterprises should route fixed decision tasks to cheaper specialized models while reserving flagship models for high-risk open-ended tasks.

Top Architecture Tech Stack
Top Architecture Tech Stack
Top Architecture Tech Stack

Introduction

AI model evaluation is shifting from "who is smarter" to "who can make reliable decisions at lower cost." Artificial Analysis's latest Decision Index v2 shows specialized decision models Drex Max v1.1 and GLiDE v1.2 scoring 73, Mammoth 27B at 72; general models Claude Haiku 5.5 (Max) at 77 and GPT-6 Luna (Max) at 76.

General models lead by only 3-4 points but at far higher cost and latency: GPT-6 Luna (Max) costs ~$1,530 per million tasks (15x Drex Max v1.1) with ~40x latency; Claude Haiku 5.5 (Max) costs ~$6,240 per million tasks (~119x latency). For enterprise developers, the signal is not that general models are useless, but that specialized decision models now offer independent cost value for fixed tasks like document review, fact extraction, compliance judgment, and tool calling.

What Decision Index v2 Measures

The new Decision Index incorporates model confidence into the decision score. Each question scored as: Decision Score = 50 × (1 + c - (p - c)^2) Where p is the model's probability for the chosen answer, c indicates correctness (1 if correct, 0 if wrong). Aggregated, this equals:

Decision Score = 50 × (1 + Accuracy - Brier Score)

Brier Score measures confidence calibration. Overconfident wrong answers are penalized more; models that defer to humans or escalate when uncertain can be safer in production.

Decision Index v2 integrates AA-Paperclip, EvalSafe-O NET-AA, AA-Omniscience-Decision, HLE-MultiChoice, AA-Symmetry, and When2Call, covering document decisions, fact extraction, knowledge mastery, reasoning, answer consistency, and tool calling. AA-Paperclip contains 1,500 resumes, receipts, invoices, and research papers, each with 5 decision questions and 10 fact extraction questions; all must be answered correctly to pass. The new version increases private dataset share and removes near-saturated tests.

Decision Index v2 benchmark composition
Decision Index v2 benchmark composition

Specialized Models Within 1 Point at the Top

Among specialized decision models, Drex Max v1.1 and GLiDE v1.2 both score 73, Mammoth 27B scores 72 — a 1-point spread indicating converging capability on fixed decision tasks.

Specialized model scores
Specialized model scores

However, similar aggregate scores hide per-dimension differences. Drex Max v1.1 achieves 97% per-question accuracy but only 71% of documents pass all 5 decision questions, and 88% pass all fact extractions — models may answer most questions right yet fail at the document level.

AA-Symmetry tests consistency across phrasing variations. Mammoth 27B scores 96, Elephant 9B scores 93; some high scores correlate with higher abstention rates because models admit uncertainty. This matters for enterprise systems: reliable decision models must not only be accurate but know when to stop, hand off, or escalate.

General Models Stronger but Costlier and Slower

Claude Haiku 5.5 (Max) and GPT-6 Luna (Max) score 77 and 76, leading specialized models by 3-4 points. The performance gap comes with massive cost differences.

Model                  | Score | Latency per task | Cost per million tasks
GPT-6 Luna (Max)       | 76    | 11.34 s          | $1,530
Claude Haiku 5.5 (Max) | 77    | 32.65 s          | $6,240
Drex Max v1.1          | 73    | 275 ms           | $102
Celeris-1.1 Decision   | N/A   | 56 ms            | $14
Cost and latency comparison
Cost and latency comparison

Mammoth 27B trails GPT-6 Luna (Max) by only 4 points but runs 118x faster. For businesses processing millions of documents daily, the small accuracy gap must be weighed against real business loss, not simply defaulting to the highest-scoring model.

Invocation Pattern Matters More Than Token Price

Cost variation among specialized models exceeds 60x: Celeris-1.1 Decision ~$14 per million tasks, Drex Max v1.1 ~$102, Clef ~$859. The difference stems not only from token pricing but from whether document input is reused. Drex Max v1.1 and Decider v1.2 compute document embeddings once for multiple questions; GLiDE v1.2 and Decider Flash v1.2 resend the full document per question.

In AA-Paperclip, GLiDE v1.2 averages ~63,000 input tokens per task versus ~9,000 for Drex Max v1.1, making GLiDE ~8x more expensive per million tasks despite only ~33% higher token list price.

Token usage comparison
Token usage comparison

Developers should optimize request structure and caching first, then token price. Reuse document context for multi-question tasks, cache reusable context, and log actual token consumption in both evaluation and production.

Enterprise Model Routing Recommendations

Decision tasks can be split into three tiers:

Low-risk, high-frequency tasks : classification, field extraction, rule matching — use low-cost specialized models.

Medium-risk tasks : document review, compliance judgment, cross-document analysis — use specialized models with good confidence calibration; escalate to human on low confidence.

High-risk, open-ended tasks : complex research, exceptional cases, decisions requiring explanation — escalate to general flagship models or human experts.

Routing systems must log model version, input tokens, cache hits, output, confidence, escalation reason, human outcome, and final business result. Without this data, enterprises only see bills, not whether models saved money or caused rework.

Chinese developers needing to integrate decision models or tools like Claude Code via compatible APIs can configure endpoints via Code80 (https://code.ai80.vip/home); model routing, data permissions, key isolation, budgets, and production audits should remain under project-side control.

Frequently Asked Questions

Have specialized decision models surpassed general LLMs?

No. General models still lead in aggregate score, but specialized models may offer advantages in cost, speed, and confidence calibration for fixed tasks.

Why include confidence in Decision Score?

Production systems need to know if an answer is reliable. A model that is wrong but highly confident is more dangerous than one that admits uncertainty.

Why does task cost vary widely with the same token price?

Resending the same document for each question repeats input tokens. Caching and document reuse strategies significantly affect final cost.

How should enterprises choose a decision model?

Compare accuracy, document-level pass rate, confidence calibration, latency, token usage, human escalation rate, and error cost on real business data — not just leaderboard totals.

When to escalate to a general flagship model?

When tasks are highly open-ended, evidence is complex, risk is high, or specialized model confidence is insufficient; log the escalation reason.

How can Chinese developers access compatible APIs?

Use Code80 to learn integration, while managing model permissions, keys, data retention, call budgets, and audit logs internally.

Original Source

Signed-in readers can open the original source through BestHub's protected redirect.

Sign in to view source
Republication Notice

This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactadmin@besthub.devand we will review it promptly.

Cost Optimizationmodel routingconfidence calibrationAI model benchmarkingenterprise AI deploymentDecision Index v2specialized decision models
Reader feedback

How this landed with the community

Sign in to like

Rate this article

Was this worth your time?

Sign in to rate
Discussion

0 Comments

Thoughtful readers leave field notes, pushback, and hard-won operational detail here.