Decision Models in Production: Threshold Tuning, Calibration & Training Essentials

Introduction

Decision models (jev-like) offer deep potential for automated evaluations but require careful tuning before production. This article builds on Hamel Husain and Antaripa Saha's "How To Use Decision Models for Evals" and adds original insights. The core engineering problem: when replacing generative LLM-as-a-Judge with decision models for automated evals, how to avoid performance loss and misjudgment from using default thresholds and raw output probabilities.

Why Decision Models?

Generative LLM judges hit two bottlenecks: (1) generation overhead — even for a binary Pass / Fail, autoregressive models must emit tokens sequentially, increasing latency and API cost; (2) no continuous, trustworthy confidence — discrete Pass / Fail outputs provide no probability information, verbalized confidence scores cluster at discrete bins (0.5, 0.8, 0.9) and are uncalibrated, and raw logprobs on answer tokens (e.g., Yes/No) collapse to 0.999 or 0.001 due to RLHF/GRPO training.

Decision models (TypeSafe Jev, Perplexity Decider, Liquid d1, Cloudflare Clef, OpenAI Decisions) remove the autoregressive loop, performing a single forward pass to read logits from a decision head and map them to continuous probabilities in [0,1]. This enables free threshold movement to match business false-positive/false-negative trade-offs and using scores as risk confidence for tiered handling.

Experimental Setup

100 real traces from an apartment rental agent system (20-turn conversations with action logs) were labeled for errors (booking failures, missed human handoffs). Six decision models returned failure probabilities:

TypeSafe Jev 1.13 (structured input)

Perplexity Decider 1.1 27B

Liquid d1 and Liquid d1-ultra

Cloudflare Clef Flash

OpenAI Decisions (GPT-6 Luna)

Models were compared on performance (F1 against human labels) and cost (API pricing per 100 evaluations). Latency was excluded due to external service factors.

Understanding Decision Thresholds

Each model outputs a score in [0,1]; APIs default to a 0.50 threshold (score ≥ 0.50 → fail). Using 0.50 blindly can be disastrous. Example: a model scores true failures at 0.40 and passes at 0.10 — perfect separation at threshold 0.30, but 0.50 catches zero failures. In reality, score distributions overlap; lowering threshold increases recall but hurts precision. The trade-off is between precision (control false alarms) and recall (control misses). Business context dictates the balance (e.g., criminal trials favor precision, cancer screening favors recall).

To compare models fairly, F1 (harmonic mean of precision and recall) was used. Score distributions for failures (orange) and passes (blue) were plotted; moving thresholds from 0.50 to per-model optimal F1 showed OpenAI Decisions separates classes most cleanly, achieving highest F1.

Setting Business Minimums

Before maximizing F1, define minimum viable precision and recall. For an automated alerting system, both ≥80% was required. Precision-recall curves for each model were plotted; the top-right rectangle (precision ≥80%, recall ≥80%) defined the qualification zone. Three models entered: Perplexity Decider 1.1 27B, Liquid d1-ultra, OpenAI Decisions. Three failed regardless of threshold: Jev 1.13, Liquid d1, Cloudflare Clef Flash.

Threshold Tuning with Cross-Validation

For qualified models, thresholds were tuned to maximize F1 subject to precision ≥80% and recall ≥80% on training folds; unqualified models were tuned without constraints to show tuning impact. A prompt for a coding agent was provided to run stratified 5-fold CV, selecting thresholds on training folds and evaluating on held-out folds, reusing saved scores without new model calls:

Using my saved model scores and binary labels (1 means failure), tune a decision threshold to maximize F1 while requiring at least 80% precision and 80% recall. Use stratified five-fold cross-validation, choosing thresholds on the training examples and measuring performance on the remaining examples. If comparing models, use the same splits for each. Only select thresholds that meet both minimums on the training examples. If none qualifies, report that rather than relaxing either requirement. Report validation precision, recall, and F1, along with any splits where no threshold qualified or validation performance fell below my minimums. Compare with the default threshold of 0.50. Reuse the saved scores without making new model calls.

Results: qualified models gained less from tuning (already near optimum), while unqualified models saw large F1 jumps but still missed the 80/80 bar. Even small precision/recall shifts matter in production. Business weights rarely match F1's equal weighting; use Fβ for precision/recall emphasis, which changes optimal thresholds and model rankings.

Performance vs. Cost: Pareto Frontier

After tuning, the three qualified models were compared on F1 and API cost per 100 evaluations:

Perplexity Decider 1.1 27B: F1 0.79, cost 0.82¢ — cheapest qualified option, extreme cost-effectiveness.

Liquid d1-ultra: F1 0.79–0.87, cost 0.82–3.72¢ — middle ground, excellent native calibration.

OpenAI Decisions (GPT-6 Luna): F1 0.87, cost 3.72¢ — performance leader but ~4.5× Perplexity's cost.

The Pareto frontier connects these points, showing the cheapest model for any desired F1 level.

Probability Calibration: Can We Trust the Scores?

Beyond binary classification, scores are often used as confidence estimates. Calibration curves (predicted probability vs. observed failure rate) reveal model reliability. Jev 1.13 and Liquid d1-ultra were nearly perfectly calibrated (on diagonal). Clef Flash, Perplexity, and Liquid d1 deviated; Clef Flash predicted ~8% failure for a bucket with 60% actual failures.

Platt scaling (logistic regression on log-odds) was applied via 5-fold CV. Perplexity, Liquid d1, and Clef Flash improved significantly; OpenAI Decisions slightly worsened. Brier score (lower better) was used to verify improvement. A prompt for automated CV calibration with Brier reporting was provided:

Using my saved failure probabilities and binary labels (1 means failure), test whether Platt scaling improves each model's probabilities. Use stratified five-fold cross-validation with the same splits for every model. Fit the calibration mapping on each training split and apply it to the remaining traces. Fit on log-odds, clipping probabilities away from 0 and 1 first. Plot predicted probabilities against observed failure rates before and after calibration. Report Brier scores (lower is better), and explain which models improved or got worse. Reuse the saved scores without making new model calls.

Judge Granularity and Iteration Loop

For production, build one judge per specific failure mode (e.g., separate judges for booking failure vs. missed handoff). Narrower tasks reduce inter-model performance gaps. Practical workflow: start with a capable large model, then try smaller/cheaper models with prompt tuning. Decision models match LLMs on classification/routing at lower cost, but LLMs still lead on knowledge/reasoning tasks (per Randy Olson).

Four-step iteration loop:

Set thresholds: define precision/recall minimums per business, eliminate models below bar.

Tune thresholds first: optimize each model's threshold on CV with business-appropriate metric (not just F1).

Verify and calibrate probabilities: if using scores as confidence, check calibration curves; apply Platt scaling only if Brier score improves.

Holistic trade-off and continuous iteration: decide based on workload performance, cost, latency. Re-run loop when prompts change.

Mathematical Insights (Advanced Analysis)

3.1 Calibration vs. Discrimination

Jev 1.13 was well-calibrated but failed the 80/80 PR threshold, while miscalibrated Perplexity passed and then calibrated well. A model's score serves two distinct tasks: (1) ranking positive vs. negative samples (discrimination/resolution), (2) reporting true failure rates per score (calibration/reliability). Murphy decomposition of Brier score: Brier = Reliability + Resolution + Uncertainty. Reliability measures calibration error; resolution measures how much true failure rates vary across score bins. A model predicting the base rate for all samples has perfect reliability but zero resolution — useless for alerting.

Post-hoc calibration applies a strictly increasing transform (e.g., temperature scaling, Platt scaling). Such transforms preserve AUROC, entire PR curve, optimal F1, and resolution; only reliability changes. Simply scaling logits widens score range but does not improve discrimination or PR curve. Training priority: learn discriminative features first, then align output probabilities with true frequencies.

3.2 Why Default 0.5 Fails: Bayes Prior Shift and Cost-Sensitive Thresholds

In logit space, default threshold 0.5 corresponds to logit 0. Platt scaling is an affine transform: logit' = a·logit + b. A key special case is prior shift: if the model learned true posterior on training distribution with base rate π_train, but deployment base rate is π_deploy, the true logit shifts by constant log(π_deploy/(1-π_deploy)) - log(π_train/(1-π_train)). Using original scores with 0.5 threshold causes systematic over/under-prediction (consistent with Jev's low mixed ECE but higher per-benchmark ECE).

Even with perfect calibration, optimal threshold depends on misclassification costs. Let FP cost = C_FP, FN cost = C_FN. For calibrated score p, expected cost of predicting fail = (1-p)C_FP, predicting pass = p C_FN. Bayes optimal threshold = C_FP/(C_FP+C_FN). Only 0.5 if costs equal. Optimizing F1 or constrained P/R also moves threshold away from 0.5. A universal model cannot bake all downstream priors and cost structures; per-deployment threshold tuning and calibration verification on labeled samples is essential.

3.3 Training Loss Shapes Probabilities: Strictly Proper Scoring Rules

To make predicted p approach true posterior, loss functions must be strictly proper scoring rules. For binary case with true probability η, Brier loss (p-η)² and cross-entropy -[η log p + (1-η) log(1-p)] have derivatives zero only at p=η, negative for p<η, positive for p>η. Under ideal conditions (sufficient capacity, optimal optimization, matched distributions), minimizing Brier or CE drives honest probability reporting.

Common modifications shift the optimum:

Label smoothing (hard label y → y(1-ε)+ε/2) pushes optimal p toward 0.5, systematically lowering high-risk and raising low-risk probabilities.

0/1 reward (reward 1 if correct, 0 otherwise) pushes p to extremes 0 or 1 — typical consequence of optimizing accuracy without calibration.

RLCD with calibration term: reward = -(p - y)² yields policy gradient equal to negative Brier gradient. Two pitfalls: (1) with full labels, direct Brier/CE gradient has zero variance; sampling RL adds unnecessary variance. (2) GRPO-style group normalization cancels the calibration term because advantage normalization removes the (p - y) factor, reverting to 0/1 reward dynamics.

3.4 Training for Simultaneous Precision/Recall Minimums

Post-hoc threshold tuning only selects points on the existing PR curve; if the curve never enters the required region, no threshold helps. Training-time strategies:

Express constraints as linear inequalities on true/false positive rates. Track constrained metrics (e.g., max recall subject to precision ≥80%) on validation folds during checkpoint selection, hard mining, and data mixing.

Schervish threshold-weighted loss: any proper scoring rule equals integral of cost-sensitive losses over thresholds with weight density w(c). Concentrate weight near target threshold (e.g., Gaussian peak at c=0.8 plus baseline uniform weight). This preserves properness (optimum still at true η) while increasing gradient weight near the threshold. However, limited model capacity causes "error migration": improving calibration near the threshold may worsen it elsewhere. A two-group toy example shows overall Brier can increase even as threshold-region calibration improves. Schervish weighting focuses calibration budget but cannot replace feature discrimination improvement; actual constrained P/R gains must be verified experimentally.

3.5 Atomic Judges and Shared Prefix: Granularity vs. Cost

Splitting a broad judge into k atomic binary judges reduces reasoning span for single-forward models. Naïve separate API calls multiply cost by k. With shared prefix prefill (shared KV cache), the long trace is prefilled once (cost C_prefill), then k short question heads are decoded in parallel with isolated attention (cost k·C_head). When C_prefill ≫ C_head, total cost approaches a single call.

Mathematical trap: attention masking isolates heads but does not imply conditional independence of real failure events given the trace. Fréchet–Boole bounds for union probability P(∪A_i) given marginals p_i: max(p_i) ≤ P(∪) ≤ min(1, Σp_i). Example: two errors each with true probability 0.3. If perfectly correlated, union = 0.3; if independent, union = 0.51; if mutually exclusive, union = 0.6. Marginal calibration alone does not guarantee calibrated union probability.

Two consumption paths:

Path A (alerting only): independently tune threshold per atomic judge; alert if any triggers. Aggregate via heuristic (max or independent union formula) then 1D Platt scaling on trace-level labels. Limitations: loses combinatorial structure, monotonic transform preserves PR curve (only changes displayed probabilities).

Path B (diagnosis + trace-level risk): fit multivariate logistic regression on all k atomic scores (logits or probabilities) with trace-level labels. Benefits: distinguishes single vs. multiple alarms, auto-downweights redundant/noisy judges, re-ranks traces improving PR curve. With sufficient data, add interaction terms for specific error co-occurrence. However, atomic judge coverage remains the ceiling; missing failure modes require adding new atomic judges.

Conclusion

Decision models return continuous, adjustable scores, separating three concerns: discrimination (PR curve), calibration (reliability curve), and threshold (business trade-off). Cheap models often just need proper tuning and calibration to be production-ready. Two open problems remain: (1) Thinking decision models for multi-step reasoning (e.g., discount promised in turn 3 missed in turn 18) — may require pre-computed intermediate values or reverting to reasoning models; (2) Adapting thresholds and calibration to distribution shift (e.g., after bug fixes or prompt changes) — no automated low-cost solution yet for detecting score drift and scheduling relabeling.