Skip to content
Jev-as-a-Judge: System One Model Beats LLM Judges on Speed, Cost & Consistency

Jev-as-a-Judge: System One Model Beats LLM Judges on Speed, Cost & Consistency

TypeSafe AI's Jev, a System One model that returns typed decisions instead of text, achieves 100% accuracy and 92–913× lower variance than GPT-5.6 and Claude Sonnet 4.6 in agent evaluation, while costing $0.00035 per call and averaging 0.44 s latency.

ThinkInAI Community
ThinkInAI Community
ThinkInAI Community

Background: Two Existing Evaluation Paradigms

Agent evaluation today relies on either code-based evaluators or LLM-as-a-judge. Code-based evaluators are cheap, fast, and reliable but limited to deterministic inputs; they cannot easily assess open-ended agent behavior where multiple valid responses exist. LLM-as-a-judge handles unstructured inputs and reasons about whether an answer satisfies a request, but LLM judges are non-deterministic, slower, and more expensive, making them a shaky foundation for trustworthy test harnesses.

What Is Jev?

Jev is a new model released by TypeSafe AI. It is not a traditional LLM; it does not generate text. Instead, it is a System One model: given a structured state (https://docs.typesafe.ai/concepts/state), it returns typed answers with calibrated probabilities. TypeSafe claims this decision-first design yields up to 200× faster inference and 400× lower cost than comparable LLMs on classification tasks.

System One models are a class of AI models designed to make fast, structured, software-consumable decisions. A System One model evaluates a state and returns typed answers and probabilities.
Autoregressive LLM vs System One model comparison
Autoregressive LLM vs System One model comparison

Jev supports three atomic question types that can be evaluated in parallel against the same state:

Choice — pick one option, return probability and confidence. Example: "Is the final answer grounded in retrieved evidence?" Response: float 0.0–1.0.

Score — rate against an ordered rubric, return probability and confidence. Example: "How helpful is this answer?" Response: 1 (not helpful) to 5 (very helpful).

Noul — return probability that a yes/no judgment is true. Example: "Which search result best describes this run?" Response: one of searched_appropriately, searched_unnecessarily, failed_to_search.

Three question types Jev can answer
Three question types Jev can answer

Experimental Setup

The authors built a weather agent using Deep Agents (v0.7.15) (https://www.langchain.com/deep-agents) and defined a test set of five weather requests in a LangSmith dataset (https://www.langchain.com/langsmith-platform). Each evaluator judged the same five captured runs on two signals: quality (continuous score) and does_pass (binary). A human rater labeled each fixed response with the same rubric to provide a ground-truth baseline. Models compared:

Jev (via langchain-typesafe==0.0.1a2)

GPT-5.6 Luna

GPT-5.6 Terra

Claude Sonnet 4.6

Each case was repeated 100 times (500 total decisions per model). LLM judges were run through LangSmith Gateway with provider defaults (no temperature, top-p, seed, or max-tokens overrides). Jev service version was not exposed in metadata.

Results

Accuracy (Binary does_pass )

Using human labels as ground truth:

Jev: 100% (500/500)

Terra: 99.8%

Luna: 96.4%

Claude: 80.0%

Accuracy comparison chart
Accuracy comparison chart

Precision (Variance on Continuous quality Score)

Observed variance across 100 repetitions per case:

Jev: mean variance 0.0000149 Luna: 433× higher

Terra: 913× higher

Claude: 92× higher

The authors hypothesize that Jev's decision-first training (calibrated probabilities, typed answers) suits bounded evaluation tasks better than autoregressive LLMs that generate text then map to scores, but note this is observational, not causal proof.

Variance comparison chart
Variance comparison chart
Per-case variance detail
Per-case variance detail
Additional variance visualization
Additional variance visualization

Cost & Latency

Jev: $0.00035 per call, total $0.34 for the experiment; mean latency 0.44 s Claude: total

$28.17
Cost comparison
Cost comparison
Latency comparison
Latency comparison

Signal Value & Online Evaluation at Scale

The authors define signal value = binary oracle agreement × binary repeatability (probability two independent calls on the same trace return the same verdict). This rewards judges that are both accurate and stable. For a production agent generating 10,000 traces/day, Jev's low per-call cost makes dense feedback feasible: teams can run more judges on more traces, catch regressions earlier, and set alerts on quality drift.

Signal value comparison
Signal value comparison

Limitations & Future Work

Narrow test scope (one agent, five tasks); transferability to other agents and production workflows is unproven.

Low cost could amplify harm if a judge is systematically wrong; human review and judge alignment remain necessary.

No causal evidence that training objective causes lower variance.

Reproducibility

GitHub repository: https://github.com/danielgshea/jev-as-a-judge

Dependencies: Deep Agents 0.7.15, LangChain OpenAI 1.6.2, LangSmith 0.12.6, Tavily Python 0.8.3, langchain-typesafe 0.0.1a2

Original Source

Signed-in readers can open the original source through BestHub's protected redirect.

Sign in to view source
Republication Notice

This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactadmin@besthub.devand we will review it promptly.

evaluation metricsvariance reductionAgent EvaluationAI agent testingLLM-as-a-judgeJevSystem One modelTypeSafe AI
Reader feedback

How this landed with the community

Sign in to like

Rate this article

Was this worth your time?

Sign in to rate
Discussion

0 Comments

Thoughtful readers leave field notes, pushback, and hard-won operational detail here.