HiRoute Open-Source: Local Agent Routing Engine Cuts Model Costs by 90% in Real Tasks

Introduction

HiRoute is an open-source, locally running Agent intelligent routing and collaboration engine developed by the Higress team. It addresses three fragmentations developers face daily: model fragmentation (different models excel at summarization, reasoning, coding, long-context), Agent fragmentation (Codex, Claude Code, other tools), and compute fragmentation (subscriptions, API keys, local services). HiRoute lets users keep their familiar Agents while routing routine work to economical models and critical stages to flagship models, and delegating independent tasks to other Agents.

Core Capabilities

Model Routing with Decision Models

HiRoute introduces a two-tier routing: task routing (which Agent) and model routing (which model per stage). Users define "saver" and "flagship" model groups. A decision model (e.g., Jev, integrated via Bailian or TypeSafe) judges task category, difficulty, and prior-stage competence. HiRoute then selects a model group based on configurable thresholds and candidate order. For example, Qwen3.8 Flash as saver, GPT-6 Astra as flagship. Thresholds and scoring prompts are customizable per task type (coding, writing, review).

ContextHold: KV Cache Preservation

During continuous tool-use loops, HiRoute's ContextHold identifies consecutive calls within the same turn, reuses the prior routing decision, and keeps the current model to maintain KV cache prefix. When context compression or rebuild breaks the full prefix, HiRoute re-evaluates at that boundary. New user messages also trigger re-decision, but mere message appends may still reuse cache.

Task Routing & Agent Collaboration

Users retain a primary Agent and delegate independent tasks (research, implementation, test) to configured Workers. Task routing answers "who does it"; model routing answers "which capability for this phase". Both layers can be used independently or combined.

Observability & Feedback-Driven Adjustment

Every session records which model executed, token usage, cost, and stage competence scores. Low scores can trigger flagship protection; absent evidence leaves unscored. Scores bind to the actual task and model, enabling traceable review. Failed candidates fail over in order; exhaustion stops explicitly. An open HTTP decision extension protocol allows custom decision services, with OpenAPI docs and a deployable Jev reference implementation provided.

Real-World Validation

Messaging System Research: >90% Cost Reduction

Task: Research six messaging systems, produce 60 verification cards each, deliver a Kafka vs SQL consistency decision memo. Three modes compared: fixed GPT-6 Astra, HiRoute mixed routing, fixed Qwen-3.8 Flash. Mixed mode assigned six research tasks to Qwen, critical memo to Astra. In two paired runs where both passed acceptance, mixed mode reduced API-equivalent cost by 92.01% and 91.39% versus full Astra. Verification accuracy: mixed 99.03%, Astra 99.72%. Critical decisions: both mixed and Astra 3/3 correct; Qwen Flash alone 1/3. Demonstrates economical models handle bulk research, flagship needed for key judgments.

HTTPX Long Task: 343 Acceptance Tests Passed Autonomously

Task: Add sync/async JSON streaming iteration to HTTPX with format handling, encoding, consumption state, and early value emission. Single prompt to native Codex, HiRoute with economical-first plan. Execution went through four stages: Qwen → Astra → Qwen → Astra. First switch at context summary; second switch when stage competence scored 0.485 (<0.5 threshold). Astra then implemented streaming parser, ran tests, fixed async iterator closure, encoding boundaries, type checks. Total time: 31 min 36 sec, zero manual prompts or patches. Final tally: 108 functional + 229 regression + 6 incremental assertions = 343 independent acceptance tests all passed . Shows stage feedback value: economical model explores, flagship implements/verifies at re-decision points.

Three-Step Onboarding

Add models : API, supported subscriptions (e.g., Codex via CLIProxyAPI), local services. Unified catalog reused across routing plans.

Create & publish route : Choose "Smart Saver", add saver/flagship models, pick decision model (built-in Jev via Bailian/OpenRouter). Advanced thresholds collapsed by default.

Connect Agent & observe : Enable model routing in Agent page, select published plan. Session view shows actual model, usage, cost; Model Performance view shows per-stage competence scores and evidence for next plan iteration.

Architecture & Principles

HiRoute Gateway built in Rust, leveraging Higress AI Gateway engineering: protocol adaptation, model capability/eligibility checks, bounded candidate failover, streaming delivery, execution observability. Desktop (Tauri), CLI, and background service share the same core. Local-first: config, routing, observability run on-device; inference and decision calls go to chosen services (cloud or local). Four product principles: (1) Task/model separation, plans reviewable and reusable with versioned snapshots; (2) Decision models judge, HiRoute executes with threshold enforcement and bounded failover; (3) ContextHold and re-decision have clear boundaries — no permanent lock-in after upgrade; (4) Adjust based on execution feedback, keep evidence for user, support custom decision extensions.

Ecosystem & Roadmap

Integrates Codex, Claude Code, Qoder, Pi, DeepSeek Harness (5 Agent types). Built-in model catalog covers 100+ providers, 700+ model configs (GPT, Claude, Qwen, DeepSeek, Kimi, GLM families) plus 7 free-tier models . Licensed Apache 2.0. Future: (1) Connect enterprise AI infrastructure via Higress AI Gateway for centralized auth, shared routing plans, cost governance; (2) Expand open ecosystem — more Agents, models, subscriptions, decision services, advancing OpenCode, Hermes, OpenClaw, giving community decision algorithms a common runtime.