Skip to content
Why Naive LLM Routing Can Cost More Than No Routing — And How to Fix It

Why Naive LLM Routing Can Cost More Than No Routing — And How to Fix It

The article explains why simple LLM routing — using a classifier to send easy tasks to cheap models and hard tasks to frontier models — often increases costs due to double inference fees, cache misses in agent loops, and brittle routing logic, and presents DigitalOcean's Inference Router with a specialized 1.5B router model, real-time candidate ranking, and session affinity to preserve prefix caching.

ThinkInAI Community
ThinkInAI Community
ThinkInAI Community

The classifier reads the request, a cheap model handles simple tasks, and a frontier model handles the rest. This is the routing pattern everyone knows — and the one that fails in production.

A single coding agent session does five different things: analyze your codebase, write new functions, fix bugs from test output, explain methods, and search documentation. These tasks vary wildly in difficulty, yet a fixed model prices them all the same.

Uber discovered the cost. They rolled out Claude Code to ~5,000 engineers in December 2025; by April 2026 the annual AI budget was exhausted. Heavy users spent $500–$2,000/month; one executive spent $1,200 in a two‑hour session. Nothing broke — engineers simply used the tool for the workloads it was designed for. Uber’s fix was a $1,500/month per‑tool cap, limiting productivity to control cost — a band‑aid.

The core problem: a docstring query costs the same as a distributed‑systems refactor.

Routing is the obvious fix, and it works. But if you build it the obvious way and drop it straight into the agent loop, it can cost more than not routing at all. The rest of this article explains why.

Routing Is the Solution Nobody Wants to Build

Send every request to the cheapest model that can handle it well. Classification, extraction, formatting, and short summarization produce nearly identical output on models 10× cheaper; frontier models are reserved for work that truly needs them.

The research is settled. RouteLLM (ICLR 2025) trained a router on human‑preference data from Chatbot Arena and tested on standard benchmarks. Their matrix‑factorization router achieved 95% of GPT‑4 Turbo quality on MT‑Bench while sending only 14% of queries to the strong model — an 85% cost reduction. These routers also generalize to model pairs they never saw during training, so the technique isn’t overfit to one setup.

The math holds on your own traffic: route 70% of requests to a $0.10/M model and 30% to a $3/M model, and the blended rate is $0.97/M, well below $3/M. A 2026 arXiv survey on dynamic routing notes that a well‑designed routing system can exploit each model’s specialty and even outperform the single best model.

The technology is proven. Teams still hard‑code a single model because the routing layer itself is the hard part.

Four Ways DIY Routing Fails

The concept is straightforward: judge intent, then forward to the right model. But implementation hides failure modes that compound in production.

You pay for two inference calls, not one. The intuitive approach puts a small LLM classifier in front. Every request now pays for a classification call plus the actual inference call. Even with a cheap model, you add a fixed tax to every request. Savings hold only if the classifier is far cheaper than the price gap between model tiers and accurate enough that misroutes don’t eat the gap.

General‑purpose models are bad at routing. Prompts like “fix this” or “make it faster” carry almost no signal on their own; intent lives in the conversation history. General models were never optimized to parse intent against a set of routing definitions. They’re prompted into a job they weren’t trained for, and coding workloads are exactly where they go wrong most often.

Routing logic rots. Add a model, rename a task, change a price tier — you edit routing code with no evaluation framework beside it. When a routing change silently degrades quality, there’s no signal. You find out when someone files a bug.

Switching models destroys your cache. This is agent‑specific and the failure mode most implementations miss. Providers cache attention state for repeated prompt prefixes. When the same token sequence hits the same model, the cached state is reused and those tokens bill at a fraction of normal price. If you switch models mid‑session, the new model has no cache for that conversation; every token is recomputed at full price.

Infrastructure‑Level Routing

All four problems share the same root cause: routing lives in your application, so the classifier, logic, maintenance, and cache behavior are all your responsibility.

DigitalOcean’s Inference Router moves all of that to the infrastructure layer. You describe tasks and the models allowed to serve each task; the platform handles the decision on every request.

The routing engine is Plano , an open‑source AI‑native agent. Every decision runs in two phases.

Phase 1: Parse Intent

A small language model reads the conversation, matches it against your natural‑language task descriptions, and returns a routing decision.

If this sounds like the double‑inference problem discussed earlier, the difference is what executes the classification.

Katanemo’s first routing model, Arch‑Router (1.5B parameters) , was fine‑tuned for one job: read the conversation, compare it to routing descriptions, and output JSON. Here’s how it performs against frontier models on that task:

文章图片 4
文章图片 4

A 1.5B model beats Claude 3.7 Sonnet on accuracy while running 28× faster .

That’s the gap between prompting a general model to classify and running a model built for classification. Routing doesn’t need long‑form generation, tool use, or multi‑step reasoning, so the capability surface is small enough for a tiny model to master.

And it runs inside the agent, not as a second API call you pay for. The cost is ~200 ms added latency, not a second line item on your bill.

The model currently in production is Plano‑Orchestrator , trained on harder conversational patterns: ambiguous follow‑ups, mid‑conversation topic switches, and messages that shouldn’t be routed at all. It slightly outperforms GPT‑5.1 and Claude Sonnet 4.5 on overall routing accuracy, with the largest gains in coding scenarios where intent is most ambiguous.

Coding is the domain where intent is most ambiguous. “Fix this” and “try again” only make sense in light of the preceding turns, and a model trained on that pattern understands it better than a general model prompted to classify.

文章图片 5
文章图片 5

Phase 2: Rank the Candidate Pool

Once the task is known, you narrow to a candidate pool of up to three models. Phase 2 picks from that pool.

You can statically rank the pool in config and always pick the first. That works sometimes, but not reliably.

Within a single day, provider latency varies 2–3× with load.

The fastest model at 2 AM is often the slowest at 2 PM.

Prices change, latency drifts, rate limits affect availability — so a config written a month ago describes conditions that may no longer exist.

The ranking engine pulls cost data from DigitalOcean’s pricing API and latency data from Prometheus, then ranks the candidate pool by the task’s strategy:

Cost Efficiency — sort by per‑token cost

Speed Optimization — sort by time‑to‑first‑token

Manual Ranking — return your exact specified order, no re‑ranking

Optimal — use DigitalOcean’s benchmark‑based ranking

A background loop refreshes these metrics and writes them to an in‑memory cache, so request‑time reads are virtually free.

Every router also carries a fallback list. If the selected model is down, rate‑limited, or unavailable, the router fails over to the next candidate by strategy, then to your configured fallbacks.

Model Affinity: Why Agents Need Different Routing

Both phases above treat each request as independent. But agent loops don’t work that way, and almost no serious workload stays single‑turn.

A coding agent reads files, calls tools, reasons about results, writes code, and checks its own output. Every step needs everything that happened before — that’s why the loop works.

This changes the routing calculus, and it’s worth unpacking carefully.

Prefix Caching

Every LLM API call is stateless. Providers don’t remember your previous turn, so the agent resends the full conversation history each time.

Providers handle this with prefix‑based KV caching. When a request starts with a token sequence the model has already processed, the cached attention state is reused instead of recomputed.

Cached input tokens bill at roughly 10% of the normal input price.

The Agent Loop Is Repeated Prefixes

A 15‑turn coding session accumulates system prompts, tool schemas, file contents, and prior reasoning. By the later turns, ~90% of what you send is text the model has already processed.

This is the ideal scenario for prefix caching. Almost all input should be a cache hit.

The Router

The router evaluates intent on every turn. That means if turn 3 is a code‑generation task it goes to model A; turn 4 is a bug‑fix task and goes to model B.

Model B has never seen this conversation. The cached prefix on model A is useless to it, so all 50,000 tokens are recomputed at full input price.

Three problems hit at once:

Cost: In a 15‑turn loop where 90% of input is cacheable prefix, model affinity saves 45–80% on input tokens. Switching models means zero cache hits; every turn pays full price.

Behavioral consistency: Different models emit different output styles and tool‑call formats; mid‑stream switches break the agent’s parser.

Coherence: The reasoning thread is handed to a model that structures its thinking differently.

The fix is session pinning . The router makes a fresh routing decision on the session’s first request, then pins all subsequent requests in that session to the same model.

DigitalOcean makes this simple. Using X-Model-Affinity with a session ID, the router makes a new routing decision on the first request, then pins all later requests with that ID to the same model.

curl --location 'https://inference.do-ai.run/v1/chat/completions' 
  -H "Authorization: Bearer $MODEL_ACCESS_KEY" 
  -H "X-Model-Affinity: session-001" 
  -d '{
    "model": "router:test-router",
    "messages": [
      {"role": "user", "content": "Write a Python function for binary search"}
    ]
  }'

The second call with the same affinity ID skips routing and returns the same model, with "pinned": true.

You pay for the routing decision once; every subsequent turn enjoys the cache benefit.

Without this, routing inside an agent loop costs more than not routing at all.

文章图片 6
文章图片 6

Building an Inference Router

Setting up a router takes about five minutes. Here’s the full flow.

Preset Routers

The fastest start. DigitalOcean provides pre‑configured routers for Software Engineering, General, Writing, and Knowledge Base & Document Intelligence.

文章图片 7
文章图片 7

Start with Name and Description

The name becomes how you reference the router later.

文章图片 8
文章图片 8

The description field matters more than it looks. It becomes the routing prompt — one of the things the model reads when deciding where a request goes.

Add Your Tasks

A router is a set of tasks plus a fallback list. Each task pairs a description with a pool of models allowed to serve it.

You can use DigitalOcean’s preset tasks or write your own.

Preset tasks include the Coding & Development collection: Bug Fixing, Code Generation, Performance Optimization, System Architecture & Design. There are also General tasks (summarization, extraction, translation, classification) and a Knowledge Base collection for long‑document QA and RAG evaluation.

Or Define a Custom Task

Custom tasks let you set name, routing description, model pool, and priority strategy: Cost Efficiency, Speed Optimization, or Manual Ranking.

文章图片 9
文章图片 9

The description carries weight because the routing model matches against it. Specific, noun‑heavy descriptions beat vague ones.

Choose Your Models

The model selector shows live per‑token pricing, so the price gap is visible. DeepSeek V4 Flash costs $0.08 per million input tokens; Claude Opus 5 costs $5.00.

That means the cheapest and most expensive models in the same catalog differ by 62× on input cost — the entire reason routing pays off.

Add Fallbacks and Create

Fallbacks handle anything that doesn’t match a task, in the priority order you set.

Compare Your Router

The Playground runs your router side‑by‑side with a single model so you see trade‑offs on real prompts, not just benchmarks.

I gave both a hard prompt: design a transactional outbox pattern for Postgres + Kafka, including schema, poller publisher logic, and idempotency handling.

Embedded video
Embedded video

0:31

Left side: raw Claude Opus 5. Right side: a custom coding-router .

This router matched the prompt to System Architecture & Design and sent it to glm‑5.2 . Both answers were correct and complete.

The numbers:

文章图片 11
文章图片 11

94% cost reduction, 77% faster time‑to‑first‑byte.

Once traffic runs, two things are worth checking. The Analyze tab reports model match rate and fallback rate; high fallback rate means your task descriptions need tightening.

文章图片 12
文章图片 12

Router Evaluation runs your router on an uploaded dataset and scores completeness and correctness via LLM‑as‑a‑judge.

That’s how you verify a routing config before it touches production traffic.

Where This Leaves You

Token prices keep falling, but agent token consumption grows faster. Gartner estimates agent workflows consume 5–30× more tokens per task than standard chat interactions.

Spend caps solve this by removing capability. Routing solves it by matching every request to the model that can serve it, keeping capability intact.

The engineering pieces that made routing hard to build yourself are exactly what DigitalOcean has absorbed:

A routing model that beats frontier models on intent parsing

A ranking mechanism that reflects live provider state, not last month’s config

Session pinning that keeps agent loops on one model so caching actually works

The remaining question isn’t whether routing works — it’s how much of your traffic has been overpaying all along.

Original Source

Signed-in readers can open the original source through BestHub's protected redirect.

Sign in to view source
Republication Notice

This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactadmin@besthub.devand we will review it promptly.

cost optimizationsession affinitymodel routingLLM routingprefix cachingagent loopsDigitalOcean Inference RouterRouteLLM
Reader feedback

How this landed with the community

Sign in to like

Rate this article

Was this worth your time?

Sign in to rate
Discussion

0 Comments

Thoughtful readers leave field notes, pushback, and hard-won operational detail here.