Laya vs Nimble vs Kev: Choosing the Right Open-Source Jev for Agents

Where Jev Fits in the Agent Execution Chain

A customer-support ticket illustrates the split: understanding long text, planning steps, and generating replies remain general-LLM strengths. During execution, however, a series of narrower semantic questions arise:

Which queue owns this ticket?
Does it require human escalation?
Can the retrieved policy support this refund?
Should the tool result stay in context?

Traditionally these are answered by prompting a general model to generate text, then parsing labels, booleans, or scores — adding a generation step, format parsing, and retry logic. Inserting a Jev-style decision model changes the chain:

Current state
  ↓
General model: understand, plan, generate
  ↓
Jev or open-source decision model: choose, score, judge
  ↓
Policy code: compare thresholds, combine conditions, pick branch
  ↓
Tool or human: execute next step
  ↓
External result written back to state

The decision model does not take over the Agent Loop; it maps state to a constrained output. Permissions, tool execution, retries, rollbacks, and human confirmation remain in code and Harness.

Narrower Questions Yield More Usable Results

All three open-source projects center on three question types: choice — pick one from candidates (e.g., billing, technical, sales queue). score — rate on an ordinal scale (e.g., urgency: "can wait", "this week", "today"). noul — probability a proposition is true (e.g., "this command is destructive", "this ticket needs human"). A 0.5 means no clear lean, not "medium intensity".

The narrower the question, the easier it is to wire into code. The model emits no JSON and no rationale; the program reads the answer and probability and decides whether to proceed. Think of it as a semantic switch: a regular switch already has an enum value, while the decision model handles the case where code holds text but cannot decide meaning by string comparison. The branch logic stays in code; the model only maps state to a finite option.

Boundary: calculations, date comparisons, permission checks, null checks — code handles these reliably. Decision models suit natural-language or semi-structured states where answers can be enumerated but rules cannot cover every user phrasing.

Laya: Easiest Local Foundation

Laya uses lightweight discriminative models with a clear goal: make local inference and fine-tuning easy to start.

Three checkpoints: English laya (ModernBERT-large, ~421M params), multilingual laya-multilingual (mmBERT-base, 100+ languages), and laya-typed-decisions for typed-decisions workflows. A Router detects language first.

Requires Python 3.10+. After pip install laya, initialize Router, pass state and question definition, get structured result. Also provides laya-serve, MCP, LangChain/LangGraph, and ONNX integrations.

Reported single-question latency: on T4 — multilingual 32.8 ms, English 39.5 ms; on GB10 (Datawhale) — Chinese multilingual 8.31 ms, English 16.41 ms. Hardware, runtime, and test method differ; treat as order-of-magnitude only.

Distinguish base vs. fine-tuned models. In the README typed-decisions eval, fine-tuned checkpoint accuracy = 0.766; English base ~0.362, multilingual base ~0.352, random guess ~0.318, majority-class baseline ~0.461. The 0.766 comes from task-specific training, not zero-shot after pip install laya.

Many candidates cause trouble. Banking77 example: dozens or 77 intents share limited token budget, label descriptions crowd each other. Repo suggests increasing head_max_len or using vector retrieval to narrow candidates first.

Probabilities cannot be used directly as gates. Multilingual checkpoint needs self-calibration; current action.act_probability carries almost no usable signal. Keep probabilities and confidence, fit thresholds on your own samples, route the gray zone to a stronger model or human.

Laya suits teams with data, GPUs, and willingness to fine-tune and calibrate once — keeping high-frequency micro-judgments in-house. If you need zero-shot across many domains without maintaining models and thresholds, cloud Jev is simpler.

Nimble: Training Focused on "What Fact Flips the Answer"

Bespoke Nimble is built on Qwen3.5-9B with LoRA fine-tuning. Its data construction method — contrastive data curation — is especially worth examining.

Create two nearly identical samples, change one key fact to flip the correct answer. Example: policy says only Mira can authorize refund for account 42; change signer from Mira to Noah, answer flips true to false. Context otherwise unchanged; model learns which evidence actually changes the conclusion.

Latest repo releases 2,676 training samples and 324 frozen eval samples (162 near-neighbor pairs from 6 sources, synthetic reference labels). Bespoke-Nimble-9B matches reference labels at 90.12%; unfinetuned Qwen3.5-9B at 66.36%; Jev 1.13.0 at 93.21% (same README).

This shows the training method works on this eval set, not direct online accuracy. Eval scale and domain are limited; repo urges retesting on your own data. Probabilities need separate handling: latest checkpoint defaults to T=1.0, early versions used 2.179, later versions applied temperature fitting for old checkpoints. Any model change means re-testing thresholds.

Inference constraints (from README): structure definitions must be flat, fields only enum or boolean, no inter-field dependencies; max 255 options per field, max 8,192 tokens per prompt; text input only, no explanations, text, or nested JSON generation.

Output mirrors Jev: each candidate answer maps to a token, scorer reads logits, Python assembles structured result. No JSON generation, no reasoning. Mac uses Apple Silicon MLX; Linux needs NVIDIA GPU with BF16 support. 9B weights ~18 GB; merged LoRA + runtime need extra memory — weight size ≠ minimum machine config.

Valuable for researching decision-model training (public data, training recipe, serving code). For production routing across dozens of categories, first calculate 9B memory needs and structural limits.

Kev: Model Family, Isolation, and API as an Experiment Platform

Mainline: Kev-0.8B, Kev-4B, Kev-9B — all based on Qwen3.5. Qwen2.5-0.5B prototype and old Qwen3 versions kept in history.

Provides TypeSafe System One-compatible API. Start local service, point

typesafe_sdk

base_url

to localhost. Requires Python 3.12/3.13, uv sync --extra serve, service at /v1/systemone accepting choice, noul, score.

Key architecture: state encoded once; each question sees only state and its own options, never other questions' answers. Qwen3.5 DeltaNet uses independent rows and caches for isolation; pointer head computes logits over candidate representations. Service assembles structured response; model never generates JSON.

Multiple questions per request run in parallel without "leakage". Playground includes isolation, option-order, and merged-vs-split request comparison experiments, plus a chess demo: board state = state, legal moves = choice, position evaluation = score.

README results on new-source data: Kev-0.8B dev/test accuracy 0.652/0.684; Kev-4B 0.802/0.835; Kev-9B 0.822/0.852; Jev dev accuracy 0.857. Not a strict same-condition showdown — training info and data not fully consistent. Kev's value: model, training scripts, and frozen eval suite are all reproducible.

Probability calibration: current models ship with temperature calibration. Kev-9B calibration error on new-source data drops from 0.106 to 0.042. Yet at 5% error budget, auto-handled proportion ~0.45–0.57, still below Jev's 0.70 in README. Date calculation and knowledge questions are weak spots; KEV_DATE_FACTS=1 injects date-interval facts into state for model judgment.

Kev-4B is the practical starting point: runs on 32 GB Mac, balanced accuracy/resource; Kev-9B for higher accuracy/calibration; 0.8B for verifying pipeline or minimizing resources. Repo bundles model, service, and eval — easier to keep the same interface when swapping models later.

How to Choose Among the Three

Using the "double-charged, no refund yet" ticket as a test case:

Start local → Laya. Simple install, small model, Router and question templates provided. Base model zero-shot limited; Chinese probabilities need recalibration; many candidates require candidate reduction or fine-tuning.

Research data construction & 9B model → Nimble. Contrastive samples, LoRA training, logits scoring all public; good for reproduction and domain customization. Cost: high memory, structural limits, 324 frozen evals don't replace online validation.

Need System One-compatible local service → Kev-4B. Three sizes, interface, calibration eval, isolation experiments in repo. Cost: heavier service; knowledge/date/new-domain generalization still need your own tests.

Local Startup

Laya quick environment check:

python3.12 -m venv .venv
source .venv/bin/activate
pip install laya

Then Router(device="cuda", preload=True) loads weights; pass ticket text and question definition to predict. First run downloads model; pre-warm for resident service. If language switching and VRAM limited, lock to multilingual or English branch.

Kev startup commands:

git clone https://github.com/jaredpalmer/kev.git
cd kev
uv sync --extra serve
uv run --extra serve python -m kev.serve --run jaredpalmer/kev-4b --port 8009

Client points base_url to http://127.0.0.1:8009. Verify with a test request:

curl -s localhost:8009/v1/systemone \
  -H 'content-type: application/json' \
  -d '{"state":"I was charged twice.","model":"kev-latest","questions":{"billing":{"type":"noul","instructions":"Is this about billing?"}}}'

Upper layers keep using System One question definitions; switching between Kev, Jev, or other implementations mainly changes service config and eval results.

Nimble deployment adds steps: Python 3.12 env, download and merge LoRA and base, choose MLX or CUDA scorer per machine. 9B weights ~18 GB; merge phase needs extra CPU RAM and disk. Loading on a laptop only means the experiment can start, not that production concurrency is ready.

For high-risk actions (refunds, deletions, production changes), I would not auto-approve just because the model returns 0.95. Probability works better as a routing signal: high confidence + reversible actions → auto-continue; middle zone → stronger model or human; low confidence → back to info gathering and deterministic rules.

Four Gaps Between Local Demo and Agent Harness

Scope context to what the question truly needs. Each question receives only the state it requires; ticket routing doesn't need the full conversation, all tool outputs, or user privacy. This improves explainability and makes eval-sample construction easier.

Record inputs and versions. At judgment trigger: state, question definition, candidate set, model version, probability distribution, threshold, and final action — all written to the run log. When errors surface, you can reconstruct exactly what the model saw.

Encode policy in code. p > 0.8 is just a condition; reversibility, second confirmation, retry after external state change — all must be explicit in code. Model handles semantic mapping; permissions and execution are controlled by Harness.

Pre-write failure loops. Model timeout, abnormal return, probability in gray zone, tool execution failure — all will happen. Fallback to rules, general model, human, or preserve original context — each should be a pre-defined branch.

Context-compression projects expose boundaries clearly: Jev can judge whether old tool results are worth keeping, but how the message list is rebuilt, whether compression actually saves budget, and whether to retain original text on failure — these remain code decisions. The model's judgment narrows, while runtime responsibilities become clearer.

Can Open-Source Jev Replace the Official Service?

Not a simple "yes" yet.

If you need local deployment, data residency, or plan to fine-tune on your own tickets and audit data — all three projects publish model weights, training scripts, service interfaces, and eval code — enough to begin experiments.

If you require zero-shot multi-industry coverage, stable probability thresholds out of the box, and no desire to maintain model services and calibration pipelines — open-source cannot yet replace official Jev. The three repos' eval data and conditions differ; out-of-distribution performance, candidate counts, probability calibration, and machine resources must be validated on your own tasks.

From these implementations we see that judgment models can indeed be deployed and evaluated separately. Laya lowers the barrier to local fine-tuning; Nimble publishes contrastive data and training methodology; Kev bundles model, service protocol, and isolation experiments in one project.

At launch, the model occupies only one slot in the runtime. Tool permissions, human confirmation, state persistence, and failure fallback remain the responsibility of code and Harness. Whether open-source Jev becomes a replaceable, auditable layer in your system ultimately depends on those runtime details.

References

TypeSafe AI: Introducing System One Models & Jev (https://typesafe.ai/blog/introducing-system-one-models-and-jev)

TypeSafe AI Docs: Primitives (https://docs.typesafe.ai/primitives)

Laya GitHub (https://github.com/NandhaKishorM/laya)

Laya Model & Docs (https://huggingface.co/convaiinnovations/laya)

Bespoke Nimble GitHub (https://github.com/bespokelabsai/nimble)

Bespoke-Nimble-9B Model (https://huggingface.co/bespokelabs/Bespoke-Nimble-9B)

Kev GitHub (https://github.com/jaredpalmer/kev)

Kev Model Collection (https://huggingface.co/collections/jaredpalmer/kev-6aad9d0ea49f2589665e07cd)

Akshay Pachaar: Jev Clearly Explained (https://x.com/akshay_pachaar/status/2101037514945597645)

Datawhale: Open-Source Jev Local Deployment Tutorial (2026-09-23)

JGX: Jev Deep Dive: From Generative Model to Semantic Decision Engine (2026-09-22)