Cloudflare's Clef: Open-Source Decision Model Skips Token Generation for Agent Routing

Why Not Everything Needs a Generative LLM

A customer-service message like "Payment page error, all orders failing" requires two distinct actions: routing to the right department with urgency level, and writing a natural-language reply. Generating the reply needs a language model, but routing only needs a definite choice among known options — technical, finance, sales, urgent or not. Cloudflare's Clef and Clef-flash are built for this classification-and-routing step. You supply the message, the question, and the candidate answers (a schema); Clef returns a probability for each option. Code can then dispatch the ticket automatically, while a general LLM still crafts the reply.

Released October 1, Clef is based on Qwen3-8B-27B, Clef-flash on Qwen3-5B-9B. Both are Apache 2.0 licensed and callable via Cloudflare Workers AI. They still understand language, images, and context, but the set of valid answers is provided at request time — no retraining when a new department is added.

How Clef Works: Prefill-Only with a Decision Head

Standard LLMs have two phases: prefill (reading input) and decode (generating tokens one by one). Even to output a single JSON field, the decode phase runs. Clef stops after prefill. A specialized "decision head" extracts evidence from the model's internal representations and scores all legal options in parallel. This skips the token-by-token generation entirely.

Multiple questions share the same evidence pass. The decision head first identifies what information each option needs, lets fields exchange signals, then re-reads the input. For example, "all orders failed" simultaneously influences both department and urgency. The program receives typed answers with probabilities — no free-text parsing needed. However, controllable format does not guarantee correctness: if the candidate list omits the right department, the model cannot fix the business design.

Probability Calibration: Confidence Scores Must Be Verified

A reported 90% confidence is only trustworthy if, over many similar judgments, the model is right about 90% of the time. If actual accuracy is 50%, the system cannot auto-approve based on that number. Cloudflare adds Brier loss during training to penalize probability-outcome gaps, and uses RLCD (Reinforcement Learning from Contrastive Distillation) to optimize whole-record judgments with partial rewards for near-correct options. The goal is to reduce confident wrong answers. Still, when deployed to your business, probabilities must be re-validated.

An escalation rule can then be set: high-confidence, verified predictions auto-route; low-confidence ones escalate to a stronger model or human. The threshold depends on the cost of missing an urgent ticket — not a generic leaderboard number.

Benchmarks, Latency, and Cost: Self-Reported Data Requires Business Validation

As of October 2, official docs show:

Clef (27B): $0.24 per million input tokens, 65,536-token context, median latency 209.3 ms (vendor internal eval).

Clef-flash (9B): $0.09 per million input tokens, same context, median latency 38.8 ms.

At 1,000 input tokens per request, 100,000 flash calls cost roughly $9 in model fees alone — excluding fetching, storage, training, and the tokens for questions and option descriptions. Retries or fallbacks to larger models increase the bill.

Skipping decode shortens inference, but network distance and server config affect real latency. The 38.8 ms median is from Cloudflare's internal test, not a universal SLA. The official quality-vs-latency chart marks Clef data as "self-reported" — useful for picking test candidates, not for production acceptance.

Scores come from the Decision Index (Jev-related) and Typesafe's business workflow eval sets, run by Cloudflare. Using another company's test suite does not equal independent certification. Flash is not a drop-in faster Clef: on CLINC150+OOS intent classification (macro-F1), Clef scores 97.4%, flash 66.8%; Jev still leads on When2Call tasks. Model choice must match your specific task, not just parameter count or speed.

For security classification, Cloudflare's internal domain pipeline (fetch, render, classify) took 2.2 seconds with Clef vs 4.7 seconds with gpt-oss-120b — showing potential speedup, not proven doubling of malicious-site detection. An HN user (agrippanux) reported Clef 2-3x slower than Jev on chat moderation and missing more hate content, but without full config and data the result is unverified. The author emphasizes testing on your own ticket batch before switching.

Fine-Tuning Loop: From Logs to Custom Models via Cloudflare's Platform

Weights are open for self-hosting and modification; managed API calls are a paid alternative. Cloudflare adds a fine-tuning pipeline: AI Gateway collects requests and responses, structures tasks and rewards; Workers AI runs trial inferences; Containers' RL sandbox executes, replays, and scores; Trainer updates weights; Workers AI and BYO Model redeploy. The sandbox is an isolated trial environment — correctness is judged by your scoring rules, not the model's self-reported confidence.

Logs are not ground-truth labels. If an old system misrouted a ticket and that error goes into training uncorrected, the new model learns the mistake. Data curation and reward design are the critical, non-skippable steps. Currently Cloudflare's Forward Deployed Engineers (FDE) assist customers with fine-tuning; a self-serve platform is still under construction. The diagram is a proposed flow, not a one-click product today.

This layout suggests Cloudflare monetizes managed inference, training, and deployment while developers can still download Apache 2.0 weights. No full price list is published yet. Cloudflare promises managed Clef does not read, store, or train on request/response data by default; fine-tuning and data collection require explicit opt-in. The open model supports images and video; the managed API currently documents image support. To try, use model IDs @cf/cloudflare/clef and @cf/cloudflare/clef-flash, compatible with Jev/System One API.

Practical Takeaway: Test on Your Own Workload First

Pick one high-volume routing point, compare historical tickets against the current pipeline. Measure both accuracy and miss rate, plus p95 latency and total cost. Only when the multiple-choice step is stable enough, let code auto-execute; keep reply writing, task planning, and exception handling with the general LLM.