Jev vs BERT: Why Decision Models Beat Classifiers for Agent Routing

What Is Jev

Jev is TypeSafe's first publicly available decision model. Unlike LLMs that generate free-form text, Jev takes a question and a declared set of allowed answers, then returns a single choice plus a probability and confidence score. The author likens it to a judge holding up a card: "Urgent, 87% confidence" instead of a verbose analysis.

Multi-Agent Routing Requires Many Small Decisions

In a multi-agent system, a single user message triggers several classification tasks: which agent handles it, emotion level, public-opinion risk, need for human escalation, and result verification. Each task picks from a small label set. Running a separate LLM call for each generates excessive tokens and latency.

Example: E-Commerce Customer Service

"I bought headphones yesterday, left ear has no sound, and your service attitude is terrible. I'll complain on Weibo if not handled."

This one message requires four simultaneous decisions:

Target agent: after_sales / complaint / logistics / human

Emotion level: calm / unhappy / angry

Public-opinion risk: true / false

Human escalation: true / false

Three Limitations of BERT-Era Classifiers

1. Labels Are Fixed at Training Time

Adding a new "invoice" agent means collecting labeled data, retraining, re-evaluating, and redeploying. Until then, "I want an invoice" gets forced into an existing category like "consultation". Jev accepts the routing set at inference time; adding a route is just adding a line of description.

2. One Model Per Question

BERT typically needs a separate model or head for each classification task, each with its own data, evaluation, and deployment pipeline. Jev answers multiple typed questions (choice, score, boolean) in a single forward pass.

3. Inputs Are No Longer Single Sentences

Routing decisions often depend on multi-turn dialogue mixed with tool JSON output, e.g.:

User: Where is my order?
Agent(logistics): Package stuck at transit center.
Tool return: {"status": "exception", "reason": "address_invalid"}
User: Didn't I update the address yesterday??

This context exceeds the "single sentence in, single label out" assumption.

How Jev Addresses These

1. Natural-Language Descriptions for Dynamic Options

Each route carries a free-text description. Official example:

deterministic_code: "Deterministic logic can handle"

fast_llm: "Short text generation, low reasoning demand"

reasoning_llm: "Requires multi-step understanding or synthesis"

human_review: "Ambiguous, sensitive, or beyond declared scope"

Adding an invoice route only requires:

invoice_agent: "User asks about invoice issuance, title, reissue, or void"

No retraining needed. The same model handles different decision tasks.

2. Multiple Questions in One Request

Same input, multiple typed questions at once:

Input: <user message + conversation context>

Question 1 (choice): Which agent? Options: after_sales / complaint / logistics / human
Question 2 (choice): Emotion level? Options: calm / unhappy / angry
Question 3 (boolean): Public-opinion risk?
Question 4 (boolean): Human escalation needed?

Structured response example:

Question 1: complaint, probability 0.82
Question 2: angry, probability 0.91
Question 3: true, probability 0.76
Question 4: true, probability 0.68

Downstream code consumes this directly without parsing natural language.

3. Calibrated Probabilities Enable Threshold Gating

Jev returns both a choice and a confidence score. Systems can set policies:

Confidence ≥ 0.9 → route directly to agent

0.6–0.9 → send to reasoning LLM for second opinion

< 0.6 → escalate to human

In the complaint example, "human escalation" confidence 0.68 falls in the middle band, triggering a reasoning-LLM review or direct human handoff due to public-opinion risk.

Why Not Just Use an LLM for Routing?

LLMs generate explanatory text that must be parsed, risking "text drift" (e.g., "angry" becomes "quite upset"). Rough cost estimate (hypothetical): 30 decision steps × 100 output tokens × 1 second = 3000 tokens and 30 seconds latency. A decision model returns only the choice and probability, cutting latency and tokens by an order of magnitude. One benchmark reported ~495 ms for a multi-question request on Apple M5 (personal test, environment-dependent). Target p50 latency: <100 ms excellent, 100–300 ms acceptable, >300 ms needs quality justification.

The Real Challenge: Probability Calibration

Accurate probabilities (calibration) are critical. If a model says 90% confidence, roughly 90% of those predictions should be correct. Overconfident models (e.g., 90% claimed, 70% actual) cause unsafe automation. Traditional softmax scores are often uncalibrated. Jev reportedly uses RLCD-style training to make reliable probabilities a training objective. The principle: knowing when the model will be wrong matters more than squeezing extra accuracy.

Production Architecture: Three-Layer Division

Request arrives
  │
  ├─ Hard policies (permissions, prohibitions) → code, before any probabilistic decision
  │
  ├─ Decision model (fast)
  │       ├─ High confidence + simple → direct handler
  │       ├─ Low confidence / long-tail → reasoning LLM (slow)
  │       └─ High risk / ambiguous → human
  │
  └─ Handler owns permissions, validation, output quality

Pseudocode:

if risk_class == "prohibited":  # hard policy before probabilistic routing
    return reject()

decision = decide(
    state=request_state,
    question="Which handler?",
    choices=ROUTES,
)

if decision.confidence >= THRESHOLD:
    return run_handler(decision.choice)
else:
    return escalate(request_state)  # reasoning model or human

Three Requests, Three Paths

"Where is my order?" → logistics, confidence 0.97 → direct to logistics agent.

Complex refund with cross-order coupon → after_sales, confidence 0.55 → reasoning LLM.

Threatens consumer complaint and social media → complaint + public-opinion risk true → human.

Most requests are simple and high-confidence, so overall cost and latency drop sharply. The decision model only picks the handler; the handler itself enforces permissions and business logic (e.g., refund amount calculation stays in code).

Caveats: Don't Overhype Decision Models

1. Business-Specific Training Still Required

Zero-shot "looks usable" is far from meeting business metrics. Real data collection, boundary clarification, hard-case augmentation, rejection mechanisms, and continuous online tuning remain essential. Example: distinguishing "product quality + unhappy tone" (after-sales) from "service attitude dissatisfaction" (complaint) requires proprietary labeled data.

2. Shadow Validation Before Launch

Run in shadow mode: production uses existing logic while decision model logs its choices. After a week, compare. If "refund" category shows 80% actual accuracy at 0.9+ confidence, raise the threshold or disable auto-execution for that category.

3. Constrained Output ≠ Correct Judgment

Output limited to declared choices guarantees format validity, not correctness. A complaint can still be misclassified as general inquiry with high confidence.

4. Not Every Scenario Needs It

Fixed 5 intents, 10k+ labeled samples, stable for six months → fine-tune a small model or BERT (cheaper, more controllable). Use decision models when: - Routing set changes frequently - Agent responsibilities shift - Single input needs multiple heterogeneous questions - Input is multi-turn dialogue + tool JSON

5. Lack of Independent Benchmarks

Available comparisons are vendor or personal tests. No public "Jev vs fine-tuned BERT" benchmark exists. Test with your own data.

Summary

BERT solves fixed-label classification. Jev targets dynamic labels, multi-question inference, and trustworthy probabilities for "decision" tasks. Production success still hinges on your data, calibration thresholds, and engineering the fast-model + slow-model + human pipeline. Count how many steps in your agent system are essentially multiple-choice questions—that overhead is often the biggest optimization opportunity.