HiRoute: Dynamic AI Model Routing with Jev Decision Model for Cost-Effective Agent Workflows

HiRoute is an open-source, locally-run routing engine that integrates with AI agents to dynamically choose between economical and high-capability models during task execution. It leverages the Jev decision model (via Alibaba Cloud Bailian System One API) to evaluate two signals at each decision point: the reasoning difficulty of the upcoming work (ordinal judgment: simple vs. complex) and the completion quality of the previous stage (assessment score 0–1). If the upcoming work demands stronger reasoning or the prior stage falls below a competence threshold (default 0.5), HiRoute routes to the main model group; otherwise it uses the cost-saving group.

Two Routing Modes

Smart Saving (ordinal only): A single branch with two model groups. Users define criteria for "simple" (e.g., extracting and formatting known facts) and "complex" (e.g., deriving new conclusions, analyzing conflicting evidence). Jev returns probabilities for each tier; the simple-tier probability is compared against a default 0.8 threshold. A concrete example shows a request where simple: 0.93 and complex: 0.07 leads to the saving group.

Custom Branches (categorical + ordinal refinement): For workflows needing finer division (e.g., writing vs. review), a categorical choice first selects the branch (writing/review). Each branch can then have its own ordinal refinement (simple/complex) with separate model groups. The article provides a full JSON decision definition for a review branch where the refinement distinguishes "scope-limited wording checks" from "full source and causal verification". HiRoute converts this into a choice question and a score question in one System One call, then reads only the chosen branch's complexity probabilities.

Historical Context & Stage Assessment

HiRoute assembles a structured state object for Jev containing: current user input, visible conversation history (with turn_id, stage_id, branch_id, user requests, and tool-activity steps), branch descriptions, and an assessment_target that pins the exact stage to evaluate (stage ID, branch ID, turn IDs, original objective, acceptance criteria, and evidence completeness). This design lets Jev understand task progression in both interactive sessions (where user feedback like "you missed the DB-commit-success/message-failure case" provides explicit signals) and unattended runs (where stage goals, visible outputs, and tool statuses such as

{"kind":"tool_activity","tool":"run_tests","status":"failed"}

serve as evidence). History truncation preserves the current input, the target stage, and essential background; a separate history_partial flag records whether earlier context was dropped.

Competence Scoring & Relay Trigger

Stage assessment uses three anchor criteria (0 = critical errors/no progress, 0.5 = usable output with clear omissions, 1 = accurate and complete). Jev returns a System One score (0, 1, 2); HiRoute maps it to 0–1 competence via score/2. In a worked example, the current task's simple probability is 0.91 (above 0.8), but the previous stage's Jev score of 0.96 yields competence 0.48, which is below the 0.5 floor. Because the target stage shares the same branch and scoring criteria, HiRoute triggers the main model group to take over the remediation.

Experimental Validation

Unattended coding task (2026-10-04): HTTPX sync/async JSON streaming implementation. HiRoute orchestrated a Qwen → Astra → Qwen → Astra relay across continuous workflow, passing 343 independent acceptance assertions in 31 min 36 sec.

Message-system research (three runs each): Mixed HiRoute routing (Qwen-3.8 Flash for bulk verification, GPT-6 Astra for key decisions) vs. all-Astra vs. all-Qwen. Mixed mode achieved 98.61% card accuracy (vs. 99.72% all-Astra, 98.80% all-Qwen) and 3/3 error-free key memos (vs. 3/3 all-Astra, 1/3 all-Qwen), while reducing API-equivalent cost by 92.01% and 91.39% respectively (including Jev decision and retry costs).

Integration & Configuration

Add a Bailian Token Plan connection in HiRoute (endpoint

https://token-plan.cn-beijing.maas.aliyuncs.com/compatible-mode/v1/systemone

, model decision-model-preview).

Create a "Smart Saving" plan, assign models to saving/main groups.

Define simple/complex criteria and stage scoring rubrics (defaults provided).

Publish plan and connect agent to HiRoute; observe model choices, reasons, and stage scores in session view.

For custom branches, use the UI to add branch names, task conditions (prompts for categorical classification), regular models, and optional main models per branch. Thresholds (simple-probability 0.8, competence floor 0.5) and all prompts are adjustable per plan or per branch.

Custom Decision Extensions

When built-in configuration is insufficient, developers can implement an HTTP extension service. HiRoute POSTs the decision definition, current input, visible history, and assessment target to the extension; the extension calls System One and returns a normalized decision (categorical choice + ordinal probabilities) and optional assessment (score + partial flag). A Python reference implementation ( jev-decider) is provided; a minimal server wraps prepare_decision() and decision_response(). The article demonstrates injecting a team_context map (e.g., definitions of "Fact Card" and "Decision Memo") into the Jev prompt so the model understands internal terminology while still using HiRoute-supplied criteria.

All code examples, API payloads, and repository references ( https://github.com/higress-group/HiRoute) are preserved from the source.