Jev: 100K SQL Classifications in 40s for $0.50, But Misses 26% of Critical Routing

Overview

TypeSafe AI's Jev is a lightweight model designed for structured semantic judgments — boolean (Noul), categorical (Choice), and scoring (Score) — that can be invoked directly from SQL. Since September 2024, multiple data platforms have integrated Jev: MotherDuck exposes prompt_jev() as a SQL function, Dataiku wraps it in a Structured Visual Agent via LLM Mesh, ClickHouse uses it in the Data Agent MNIST evaluation framework, and community extensions exist for PostgreSQL ( pg-jev), DuckDB ( duckdb-jev, recodelabs/duckdb-jev), SQLite ( sqlite3-jev), dbt ( dbt_jev), and LlamaIndex ( llama-index-jev). The article examines real-world benchmarks, use cases, and the operational gaps that remain when semantic judgment enters data systems.

MotherDuck: Large-Scale Classification Benchmarks

AG News Benchmark

MotherDuck tested Jev on 100,000 AG News articles:

Jev : 40 seconds, 89% accuracy, ~$0.50 cost.

GPT-4o mini : 19 min 45 sec, 80% accuracy.

GPT-5 nano : 17 min 49 sec, 83% accuracy.

GPT-5.6 Terra : 31 min 59 sec, 88% accuracy, $37.58 cost.

The benchmark demonstrates a different cost curve for high-volume structured judgment versus open-ended generation, but the author cautions that a clean news classification task does not directly extrapolate to complex approval or risk-control workflows.

Financial Complaints (CFPB)

100,000 consumer complaint narratives classified into seven resolution-intent labels (refund, credit-report correction, explanation, etc.) in 82.2 seconds. 65.8% of judgments had confidence ≥0.8. A 300-record sample compared against GPT-5 and Claude Opus 5.5 showed Jev label agreement of 77.3% and 74% respectively; on the 218 records where the two LLMs agreed, Jev matched 90.4%. The article stresses these are inter-model consistency figures, not ground-truth accuracy, and recommends retaining raw text and confidence scores for analyst review.

Agent Trace Analysis

MotherDuck's online Agent (querying data, running Flights, creating Dives) produces OpenTelemetry traces. Jev scores each trace on completed, failure_mode, error_cause, efficiency, main_waste. Deterministic metrics (call count, latency, tokens) stay in SQL; only low-completion or low-efficiency traces receive fine-grained attribution. This lets developers query failure-type distributions first, then drill into specific traces.

Hacker News AI Company Sentiment

From ~488,000 keyword-matched records (2019–present), a stratified sample of 65,635 was classified for true company reference (OpenAI, Anthropic, Google, DeepSeek), primary discussion focus, and negative-theme detection. Jev judged 89% of 2019–2022 "Claude"/"Anthropic" hits as unrelated to the company; that dropped to 0.8% for 2023–2026. The author notes this reflects model judgment on keyword hits, not human-verified false-positive rates, and is limited to the Hacker News community.

Cloud Platform Hiring Signals

Job postings mentioning AWS/Azure/GCP were reduced from ~4,200 to ~750 characters (title + relevant sentences) before Jev classification into primary cloud, multi-cloud/no-clear-primary, or incidental mention. In 396,000 2024–2025 postings, ~213,000 passed the filter; Azure led in 18 of 21 European countries. The article clarifies hiring demand ≠ market share, but text disambiguation reduces misclassifying incidental mentions as core requirements.

Office Policy Classification

2,000 postings classified as remote/hybrid/on-site/unspecified. Optimized regex: 0.2 sec (22 match/exclude groups to avoid false positives like "Remote Sensing"). Jev: 1.7 sec. Generic LLM: 37 sec. Jev uses category descriptions to infer context, reducing hand-written exceptions. No ground-truth accuracy comparison was provided, so speed alone cannot determine superiority.

Sales Call Intent Detection

Jev flags calls containing "need to analyze large volumes of user conversations or product reviews," then joins via account_id to CRM (customer size, contacts) to produce a review list for sales. This is a constructed demo with synthetic data; no real conversion results. It illustrates the pattern: model handles semantics, existing systems handle fields and actions.

Dataiku: Vendor Onboarding Routing Experiment

Dataiku's TypeSafe AI Plugin routes vendor requests through a Structured Visual Agent. Jev 1.13, GPT-5.6 Luna, and GPT-5.6 Sol were tested on 200 labeled requests:

Jev : 157/200 correct (78.5%), avg 0.3 sec, ~$0.03/1k.

Luna : 169/200 (84.5%), 3 sec, $0.25/1k.

Sol : 174/200 (87%), 2.2 sec, $4.35/1k.

Jev is fastest and cheapest but less accurate. Crucially, on requests that should go to human review, Jev missed ~26% (false negatives), versus ~6% for the LLMs. Example: a purchase request where the applicant noted their manager was on vacation and they would act as both approval levels — Jev did not flag the segregation-of-duties violation. False negatives in compliance routing carry higher risk than false positives (extra manual review).

Lowering Jev's exception threshold from 0.70 to 0.30 recovered 10 previously missed requests but added 4 unnecessary human reviews. Dataiku proposes a hybrid: Jev for clear low-risk, LLM/human for borderline. This hybrid remains unvalidated. Key takeaway: cost and accuracy must be evaluated separately for false negatives vs. false positives; decoupling model judgment, routing rules, and evaluation data enables threshold tuning without pipeline rewrites.

Ecosystem Integrations

ClickHouse: Data Agent MNIST

Open-source framework for evaluating Text-to-SQL agents against custom warehouses. schema_retrieval.py uses Jev to score table-question relevance (nDCG@10, Recall@k vs. reference SQL tables). schema_retrieval_agent.py feeds ranked tables to the agent. label_failure_modes.py annotates failure causes on scored trajectories. Jev is optional; the project has not yet published independent evidence that it improves overall analysis accuracy.

PostgreSQL: pg-jev

Extension provides jev(), jev_prob(), jev_choice(), jev_score() for semantic filters in WHERE and ordering. Uses batching, concurrency, prefetch, and session caching to mitigate per-row latency. In a 2,000-row test: first query ~3.5 sec, ~$0.012 model cost; cached session hits ~50 ms. Requires PostgreSQL 14–17, plpython3u, superuser — not available on all managed services. Results are from a dev environment, not a universal performance guarantee.

DuckDB & SQLite

colliber/duckdb-jev

: maps Choice/Noul/Score to ENUM/DOUBLE/STRUCT for direct downstream SQL use. recodelabs/duckdb-jev: focuses on semantic filtering, probability ordering, batching. mattn/sqlite3-jev: C + libcurl wrapper for Jev API, enabling classification/scoring in lightweight SQLite.

These are distinct community implementations, not the same as MotherDuck's official prompt_jev(). Source code and examples exist, but installation and long-term maturity need individual evaluation.

dbt: dbt_jev

Macro package for DuckDB/ClickHouse environments. Handles agent tool-call classification, sales lead entity matching, support ticket urgency scoring. Macros generate SQL; inference occurs at execution; results can be materialized for downstream models. Introduces incremental update, label drift, and model versioning concerns: re-run history when categories expand from 4 to 6? Are scores comparable after model upgrade? Implementation and examples provided, but no large-scale production validation yet.

Knowledge & Research Data

llama-index-jev

: scores candidate documents in LlamaIndex, re-ranks retrieval. Self-reported on BEIR nfcorpus (323 queries): MiniLM nDCG@5 improved from 0.340 to 0.396 with Jev. Jev Reviewer: extracts candidate answer spans from papers/supplementary materials, copies text with source locations for researcher verification and table export.

Both extend the "model selects, software records and processes" pattern beyond database aggregation.

Individual Developer Practice

Claire Vo (September interview) reported using Jev on ~1,700 PRs, ~4,500 YouTube comments, and ~200,000 classifications across ~1,100 signals in a product insights project. Cost for 1,700 PRs: ~$0.09. These are self-reported scale/cost figures, illustrating low-cost classification entering daily engineering and product analysis, but not generalizable business outcomes.

Remaining Challenges for Production Adoption

Label validity ≠ business accuracy : High confidence does not equal calibrated business accuracy. Dataiku's missed-review experiment underscores the need for custom labeled evaluation on critical error types.

Versioning & auditability : Materialized judgments must store label definitions, model version, processing timestamp, and review status to prevent historical reports mixing different standards.

Query economics : Batching, caching, incremental updates, and input trimming are as important as model unit price. MotherDuck and some extensions send text to external model services — data governance (customer, employee, contract, internal logs) requires authorization, permission, and compliance checks first.

Right tool for the job : Simple pattern matching stays with SQL/regex; complex planning and open-ended analysis still need other models and tools.

MotherDuck offers a platform integration, Dataiku provides a public plugin with comparative experiments, and many community projects are early-stage. They have not proven Jev will become a universal standard, but they have pushed a concrete pattern into code and query pipelines: databases handle computation, a specialized model handles a slice of repetitive semantic judgment, results are stored in the data system for the next query, evaluation, or business process.

References

TypeSafe AI | Jev official site: https://typesafe.ai/

MotherDuck | Introducing prompt_jev(): https://motherduck.com/blog/motherduck-supports-jev/

MotherDuck | Jev for Analytics: https://motherduck.com/blog/jev-for-analytics/

MotherDuck | 5 Jev Use Cases for Analytics: https://motherduck.com/blog/jev-analytics-use-cases/

Dataiku | Jev Inside a Dataiku Agent: https://www.dataiku.com/tech-blog/jev-inside-a-dataiku-agent

ClickHouse | Data Agent MNIST: https://github.com/ClickHouse/data-agent-mnist

PostgreSQL | pg-jev: https://github.com/realZachi/pg-jev

DuckDB | duckdb-jev: https://github.com/colliber/duckdb-jev

dbt | dbt_jev: https://github.com/smithclay/dbt_jev

LlamaIndex | llama-index-jev: https://github.com/WiktorB2004/llama-index-jev

How I AI | Claire Vo's Jev practice: https://www.lennysnewsletter.com/p/jev-for-beginners-how-to-use-it-and