Smart LLM Routing Still Saves 90%+ Despite Cache Loss on Model Switch

Core Question: Does Cache Loss Undermine Smart Routing Savings?

Intelligent routing assigns different tasks to different models. A common concern: switching models invalidates accumulated KV cache, potentially erasing cost savings. The short answer: some cache loss occurs, but it is far outweighed by savings.

Why Cache Loss Is Limited

Switches are infrequent. HiRoute's ContextHold mechanism keeps the same model during continuous tool calls within a single task turn. It only re-evaluates at user messages or context compression. In a 40-request unattended coding experiment, the router decided only at the start and two compression points; 35 of 40 requests (87.5%) reused the existing model.

Loss is one-time and amortized. Cache loss happens only on the first request after a switch; subsequent calls hit the new model's cache. With n requests per turn, the overall cache-hit-rate drop is at most 1/n. Writing/review tasks average ~7 calls; long coding runs can reach dozens.

Switching to cheaper models often costs less even with full reprocessing. At frozen Astra × Qwen prices, Qwen 3.8 Flash costs ~1% of Astra per token. For a 100k-token prefix: reading cache on Astra costs $0.10; reprocessing on Qwen costs ~$0.019 — actually saving ~$0.08. The reverse direction (Qwen → Astra) costs more ($1.25 vs $0.0015), but that upgrade buys stronger capability.

Cache only moves the savings needle by a few percentage points. In the Astra × Qwen experiment, mixed routing with zero Qwen cache hits still saved ~91%; even if every Qwen input hit cache, savings only rise to ~94%.

HiRoute's ContextHold Mechanism

ContextHold decides when a model change is allowed:

Continuous tool calls in the same turn: reuse current model → cache continues growing.

New user message: re-judge task; may continue or switch. Continuing incurs no loss; switching loses the old prefix once.

Context compression/rebuild: original choice kept if still suitable; otherwise re-judge. Since history is partially rebuilt anyway, the extra loss from switching is smaller.

This design avoids ping-pong switching that would prevent amortization.

Compression Nodes Reduce Switching Cost Further

When context length triggers compression, early history is summarized. Even without switching, the summary and later content must be reprocessed; only static parts (system prompt, tool definitions) remain cacheable (per Claude's compaction docs). Switching at a compression point therefore loses only the remaining prefix, not the full history.

End-to-End Experiment: Astra × Qwen

Three paired runs compared mixed routing (Qwen for six document verifications, Astra for key decisions) against all-Astra. Two runs met acceptance criteria with both approaches; the third run's full data is preserved in the experiment notes.

Run 2: mixed $0.147 vs all-Astra $1.84 → 92% savings.

Run 3: mixed $0.178 vs all-Astra $2.07 → 91% savings.

In these runs, Qwen cache hits were 0. Hypothetical full cache hits would lower mixed cost to $0.124 (Run 3) → 94% savings; Run 2 would move from 92% to 93%.

Conclusion: mixed routing saves 90%+ regardless of cache. Reason: cache discounts only input, while the cheap model's bill is dominated by output tokens.

Price-Drop Sensitivity

Even if Astra cuts prices proportionally, mixed routing remains advantageous until Astra drops ~95% (assuming best-case full cache hits for Astra). At current prices, all-Astra costs ~10.3× mixed; at 20% price, ~3.4×; breakeven at ~5.1% of current price.

Operational Principles

Keep model stable within a phase to accumulate cache benefits.

Re-evaluate at rebuild points (compression, new user message) to minimize avoidable cache loss.

Upgrade to stronger models when needed to meet quality standards, avoiding rework from under-capable cheap models.

Cache loss is a bounded, one-time overhead controlled by HiRoute. Assigning suitable work to markedly cheaper models saves an order of magnitude more than the combined cost of cache rebuild, routing decisions, and occasional rework.

Try It

In the "Smart Routing" page, enable "Smart Save", add a "Save Model" and a "Primary Model". After a run, open "Run Record" in the session to see per-request cache hits, actual model used, and estimated cost. Details in the Model Routing Guide.

References

[1] Model Routing Guide: https://hiroute.ai/docs/model-routing/

[2] Claude Compaction & Cache: https://platform.claude.com/docs/en/build-with-claude/compaction-threshold#maximizing-cache-hits-with-system-prompts

[3] Astra × Qwen Experiment: https://hiroute.ai/news/astra-qwen-smart-routing/

[4] Experiment Details: https://github.com/higress-group/HiRoute/blob/main/experiments/cases/research-cost-quality/README.md