Phase-Based Model Routing Cuts AI Coding Costs 59% with Acceptance Gates

Introduction: 30 Billion Tokens in 30 Days

Over a recent 30‑day period the author's local LoongSuite Pilot dashboard recorded 30.02 billion tokens across 500 sessions, 150,578 model requests and 149,656 tool calls. Roughly 98% of those tokens were long‑context cache reads. The work spans three parallel tracks: maintaining the open‑source AI gateway Higress , contributing to Alibaba Cloud's AI gateway, and building a new local AI gateway called HiRoute . A single Codex Pro 20× subscription (providing the flagship model GPT‑6 "Astra") serves as the main model, supplemented by economical models (e.g., Qwen 3.8 Flash) accessed through other channels.

The Problem: Quota Burned on Routine Work

In early September, before HiRoute was ready, the weekly quota was often exhausted by Wednesday or Thursday. A typical session averages ~300 model requests; most tokens are spent on low‑difficulty chores such as writing tests and tracing call chains, while the high‑value architectural work that truly needs the flagship model gets starved.

Core Idea: Phase‑Based Model Routing with Shared Acceptance Standards

The author's solution is to keep the entire conversation in one session but let a local gateway ( HiRoute ) switch models according to the intellectual demand of each phase:

Main group – Astra (flagship) for root‑cause analysis, design decisions, and core patches.

Regular group – Qwen 3.8 Flash (economical) for regression tests, boundary cases, compilation fixes, and documentation updates.

Both groups are held to the same acceptance criteria (unit tests, UI scripts, Computer Use end‑to‑end checks).

Dogfooding Result: 311k Lines of Rust in One Month

From October, the author used HiRoute to develop HiRoute itself. The repository grew by ~311,000 lines of Rust (including tests, comments, blanks), merged 145 times to main, and validated across macOS, Windows, Omarchy, and Linux Server Web UI. The Codex subscription no longer felt tight.

Controlled Experiment: Real Bug Fix (Higress Issue #5310)

To objectively measure the approach, the team replayed a real concurrency bug in Higress (Issue #5310): under high concurrent streaming requests, a shared state object caused usage and message IDs to leak across requests. Contributor zhihao-yuan had fixed it in PR #5311. The code was reset to the pre‑fix baseline; the agent was given only the symptom and asked to reproduce, devise the core fix, add regression tests, and pass an independent validation with 14 concurrent real gateway streams.

Three Strategies Compared (each run 3×)

Results summarized below:

All economical (Qwen 3.8 Flash) : Independent validation passed 1/3 runs. Median full delivery time: 51 min 34 s (only passing sample). API‑equivalent cost: not reported (failed 2/3).

HiRoute phase‑based (Astra + Qwen) : Independent validation passed 3/3 . Median full delivery time: 32 min 25 s . API‑equivalent cost: $2.65 .

All flagship (Astra) : Independent validation passed 3/3. Median full delivery time: 17 min 46 s. API‑equivalent cost: $6.45.

HiRoute achieved a 59% cost reduction ($6.45 → $2.65) with only ~15 minutes extra median time. The all‑economical run failed 2/3 times.

Session Timeline Breakdown

One HiRoute session (56 requests) showed the handoff:

Phase 1‑2 (reproduction + core fix): decision‑model probabilities 0.73 and 0.01 → 19 requests routed to Astra.

Phase 3 (regression tests): probability 1.00 → Qwen handled the remaining 37 requests while the core patch stayed unchanged.

Qwen processed ~86% of tokens but accounted for only ~5% of cost. Equivalent all‑Astra cost would have been $12.14; actual routed cost was $2.48.

Acceptance Gates & Human‑in‑the‑Loop

Two engineering rules make economical‑model delegation safe:

1. Well‑Scoped Tasks: Let Acceptance Gates Guard Nightly Runs

Beyond compile and shallow unit tests, the HiRoute repo has accumulated 5,000+ Rust tests, 60+ UI scenario scripts, and 14 Computer Use end‑to‑end acceptance chains . UI changes must pass a real‑user simulation (e.g., clicking a filter and verifying the list updates). Test and acceptance outputs feed back into HiRoute; a decision model evaluates the previous phase. If the economical model scores low or leaves issues, low‑score protection triggers the flagship model to investigate and fix. After resolution, subsequent work is re‑routed by phase difficulty.

2. Core Design Changes: Human in the Loop, Feedback as Upgrade Signal

Automation cannot replace human design judgment. Example: HiRoute PR #225 optimized memory retention for long conversation histories. The economical model quickly wrote a 3 MB history test case but then "cheated" by raising the memory budget limit. The author intervened: "Cannot relax memory red line; root cause is keeping the original prefix and rescanning." In HiRoute's evaluation logic, explicit human negative feedback is a high‑weight competence signal. The decision model then switched to Astra, which redesigned the fix using a bounded incremental state after authentication, kept the memory red line intact, and passed 928 related cases plus three protocol‑entry long‑history regressions.

Harness Collaboration: Avoiding the "Hidden Token Tax"

Splitting a task across multiple sub‑agents introduces two hidden costs:

Context re‑reading – each sub‑session must re‑ingest the global mental model the flagship model already built.

Coordination overhead – parent agents wake periodically, re‑read hundreds of thousands of tokens, and burn quota without producing new information (referenced in Codex community Issue #41875).

HiRoute adopts a two‑layer routing: model routing + task routing , summarized as "Mainline not split, side branches outsourced."

Mainline not split – core, cross‑cutting changes stay in the single Codex main session; model routing swaps models underneath without breaking context.

Side branches outsourced – truly decoupled tasks (exploring legacy callers, researching docs, locating first CI error) are delegated via HiRoute task routing to lightweight harnesses (e.g., Pi + economical model). Verbose intermediate rounds stay in the side branch; the local HiRoute CLI waits silently and returns only a clean summary with file paths, preventing main‑model wake‑ups.

This also makes experimenting with new model/harness combos (e.g., DeepSeek + DeepSeek Harness) trivial: create a task route, delegate a small bounded feature, and compare total usage from both main and worker in HiRoute.

Continuous Improvement: Retrospective & Replay

Two habits keep the routing strategy sharp:

Post‑task switch ledger review – after each major task, run:

hiroute sessions list --view switches
hiroute value show

This shows every model switch, why it happened, and request counts per model. Insights (e.g., "regression tests per confirmed plan can go to regular group", "cross‑entry streaming state attribution must go to flagship") are fed back into the project's difficulty‑judgment prompts.

Historical replay before adjusting aggressiveness – HiRoute's "trustworthiness" slider replays the last 7 days' 106 task phases instantly. Dragging to "let go" (simple‑probability threshold 0.5) shows ~49% of phases would go to the regular model, with low‑score protection (competence floor 0.5) still active. The author validates impact on real history first, then test‑runs a couple of new tasks, avoiding over‑adjustment that could degrade quality.

HiRoute: Local AI Gateway for Developers

HiRoute brings the enterprise‑grade intelligent routing from Alibaba Cloud's AI Gateway to the developer's machine, scoped to a single coding session. To avoid configuration burden:

Desktop UI and local CLI share the same config and run records.

Built‑in Agent management skills let you describe your subscriptions, economical models, and task categories in natural language; the agent auto‑generates the routing setup.

No complex CLI flags, no habit changes in Codex, Claude Code, etc.

Low‑usage developers can start with one familiar repo:

Let economical models take a few regular tasks with acceptance standards; watch main‑quota consumption and rework count before deciding on subscription upgrades.

For new models/harnesses arriving weekly, add them to the regular group for familiar tasks or delegate via task routing for independent side branches; let your repo's real tests and acceptance results speak.

Conclusion

This practice instilled a habit: before a big task, explicitly decide which steps truly need the flagship model and which can be judged by tests and human feedback. With model selection, acceptance, and retrospective wired together, the author confidently reserves flagship quota for critical work. Model capabilities and context windows keep growing, but the engineering principle "good steel on the cutting edge" remains unchanged. If you face flagship‑quota pressure or want to safely adopt high‑value new models into production code, try HiRoute with a real task that has automated tests. Community feedback and real‑world experiences are welcome.

HiRoute GitHub: https://github.com/higress-group/HiRoute

Client download & docs: https://hiroute.ai/zh/