TokenRouter: Efficient Token-Level LLM Routing via Async Handoffs & Delayed Batching

Introduction

Token-level routing allows multiple models to collaboratively generate a single output sequence by switching models during decoding. While algorithmic savings are promising, they often fail to translate into serving efficiency because model speeds differ, requests switch frequently, batches fragment, and cache management overhead grows. TokenRouter addresses this by decoupling routing policy from the serving runtime, providing a request-centric programming interface, asynchronous handoff with state retention, and delayed batching guided by a throughput model.

The paper is a systems paper, not a survey or new routing algorithm. It has been accepted at NeurIPS 2026 (arXiv:2610.12242). Authors: Tianyu Fu, Tengxuan Liu, Ruoxi Wang, Yixin Dong, Yi Ge, Yichen You, Yu Wang (Tsinghua University and Carnegie Mellon University). Code: https://github.com/thu-nics/TokenRouter.

Related Work

Model Routing

Existing methods operate at session or query granularity. Efficiency-oriented routes easy queries to cheaper models; quality-oriented selects expert models. They are compatible with single-model serving stacks because requests rarely migrate mid-decoding.

Token-Level Routing

Methods use trained lightweight routers or signals like confidence, entropy, prediction divergence, or learned handoff signals. Despite algorithmic differences, they share system requirements: select next model, transfer request state, accept peer-generated tokens, continue decoding. TokenRouter abstracts these into a unified interface.

LLM Serving Systems

Continuous batching, paged KV caches, and structured generation improve single-model efficiency. However, multi-model collaboration is not arbitrary model cooperation. Speculative decoding has a fixed draft-verify structure, while token-level routing may switch dynamically at any position. Forcing it into fixed synchronization creates idle waits and small batches. TokenRouter extends existing engines by letting each model manage local batching and prefix caches, while cross-model collaboration is handled by a dedicated handoff mechanism.

Programming Interface

Principle: Request-Centric Sequential Programming

Developers naturally track a single request: receive context, decode once, check result, optionally handoff, receive peer result, continue. TokenRouter lets developers describe algorithms in this sequential style without explicitly arranging multi-model batches or global synchronization. Actual execution organizes request queues and batches per model, with each model advancing independently.

Core Interface

The interface consists of three functions: route, send, and receive, answering "where to go", "what to handoff", and "how to continue".

Routing decision. route(result) runs after model forward pass, using hidden states, output probabilities, or sampled tokens. Returns 0 to continue on current model, non-zero to handoff to target model. Different algorithms reuse the same runtime, placing decision logic in this function.

State handoff. send(req) constructs a cross-model request object containing request ID, token suffix unseen by peer, completion flag, and algorithm-specific metadata. Position anchors indicate where new tokens fit in context. The interface allows deciding whether to include the just-sampled token: if current model's prediction is unreliable, the peer should regenerate that position rather than accept it as confirmed.

Receive and resume. receive(peer_req) converts incoming cross-model request to local state. Default behavior accepts incoming tokens; algorithms with different commit rules can customize.

Important distinction: token context handoff ≠ KV cache migration. Different models have separate caches and representations; TokenRouter mainly transfers incremental tokens and request semantics.

Running Example: R-Stitch

Small model checks output entropy. When entropy exceeds threshold, request is handed to large model, discarding the uncertain token so large model regenerates it. Large model hands back when low-entropy condition met. Only the unseen token suffix is sent, not the entire history each time.

Other algorithms map to the same interface: CITER triggers on low confidence, R2R uses prediction divergence, Co-LLM uses handoff probability (peer returns after one token), R-Stitch allows multiple tokens before switching, ME involves token-level ensemble weights. The unified interface covers varying switch frequencies and commit semantics without forcing a single round-trip pattern. It reduces implementation and integration cost; generation quality still depends on routing strategy and participating models.

System Design

System Overview

Each model runs in an autonomous sub-server containing a scheduler, model runner, private KV cache pool, and routing functions. External clients face a single unified entry point, unaware of internal model switches. Implementation based on SGLang 0.5.1, frontend adapted to FastAPI, sub-servers communicate via ZeroMQ in separate processes with an event loop each. The paper's "three loops" are logically decoupled execution flows, not necessarily three threads. For models sharing a device, memory budget must be jointly planned for weights and KV caches to avoid each model reserving full GPU capacity. Joint memory planning ≠ sharing the same KV cache.

Asynchronous Execution for Efficiency

Four scheduling modes compared: step-wise sync (all models finish step before next), model sync (only one model advances at a time), immediate async (no global barrier but new arrivals may miss current batch), and TokenRouter (async with batch organization optimization). Figure 2 illustrates execution timelines; example average latencies: 13.33, 13.00, 10.83, 8.67 small-model batch time units (mechanism illustration, not universal measurement).

Sub-server contains client service loop (request admission and streaming results), decoding loop (local scheduling and execution), and inter-model handoff loop (send/receive peer requests). When a request goes to another model, other local requests continue running without waiting for peer.

Key design: retain state after handoff, resume directly on return . If every return were treated as a new request, prefix matching, cache allocation, and data structure updates would repeat. TokenRouter adds a "pending return" state between "running" and "done": request exits current model's scheduling but retains service state; on peer return, system appends new tokens and resumes execution or marks done.

Performance gains come not only from concurrent model work but also from avoiding repeated request initialization. Appendix breaks down a standard serving baseline's small-model steps: model inference only 4.20%, while prefix matching, cache node locking, release, and tree updates dominate. This specific configuration shows management overhead can become the main bottleneck under frequent handoffs.

Delayed Batching for Scheduling Efficiency

Async execution alone doesn't solve irregular arrivals. If a request triggers large-model execution and others arrive subsequently, they wait for next batch; immediate execution creates many small batches. Delayed batching lets requests queue briefly until a model's queue reaches a threshold, then launches a larger batch to amortize execution cost and reduce later requests missing the batch. Letting early arrivals wait slightly may lower overall average wait, but does not guarantee every request is faster.

Thresholds are not "larger is better". Too small retains batch fragmentation; too large may cause queues to never launch, even causing models to wait on each other. Paper uses a discrete-time Markov chain to build a throughput model, with state including request concurrency, routing probabilities, model step latencies, and per-model queue lengths, current batch sizes, and remaining execution times. Model computes steady-state committed tokens per unit time, then selects threshold combination maximizing throughput from feasible set. "Committed tokens" matters: tokens generated but later discarded by a model should not count toward final output throughput.

Appendix gives feasibility condition: sum of (threshold - 1) across models < request concurrency. This avoids deadlock where all models are waiting for queues to fill while no batch is executing. Theory assumes stationary routing probabilities, request independence, and geometric distribution of consecutive execution steps. In R2R validation, routing probability 0.35, small/large model step latencies ~6.0ms and ~27.9ms. With small-model threshold fixed, theory-predicted optimal large-model thresholds matched hardware-measured optima across tested concurrency levels. This is throughput optimization under given modeling assumptions, not a universal proof for arbitrary dynamic traffic or tail-latency targets.

Experiments

Experimental Setup

Platform: single server with 8×A100 80GB GPUs (not all used in every experiment). Primary dual-model config: Qwen3-0.6B (tensor parallelism 1) and Qwen3-32B (TP=2), sharing two GPUs via CUDA MPS. ME uses 0.6B, 8B, 32B each on one GPU. Five algorithms, three workloads, 15 combinations. Main concurrency = 4; also single-request, concurrency scaling, different model combos, cross-node tests.

Three workloads: (1) Low compute inference: AIME2024, max output 2048 tokens. (2) High compute inference: AIME2024, 32B solves problems >8192 tokens, max output 8192. (3) Agent-related: SWE-Smith trajectories, input ~8192 tokens, max output 1024. Last workload tests generation service on trajectory inputs, not full online agent tool-use and task success.

Baselines: official implementations and standard serving implementations. Co-LLM official uses vLLM continuous batching per model; CITER and R2R official lack continuous batching. R-Stitch has no official code. Standard serving baseline: each model runs SGLang instance with external scheduler coordinating via single-token interface. Standard serving baseline is not "SGLang natively supporting arbitrary token-level routing at best performance"; it's a constructed baseline for comparison. Speedup explanations must state baseline capabilities and limitations.

Serving Efficiency

Figure 3 shows throughput and end-to-end latency at concurrency 4 for five algorithms across three workloads. TokenRouter achieves 2.01–64.15× decoding throughput improvement over the stronger available baseline per setting; relative to standard serving, end-to-end latency improves 2.03–63.64× . Ranges correspond to different metrics, not interchangeable, and max ratio not stable gain for every scenario.

When output limit increased from 2048 to 8192, TokenRouter throughput remained stable while standard serving baseline lost 58.1%–85.2% throughput. Longer outputs increase cumulative cost of model round-trips and state management, making handoff recovery crucial.

Paper also tests each algorithm with its original model and task settings (not just unified model combos). Figure 4 shows results. R2R (DeepSeek-R1-Distill-Qwen-1.5B/32B on AIME): throughput from 89.62 to 244.56 tok/s, latency from 751.15s to 270.19s. CITER (Qwen2-1.5B/72B on CommonSenseQA): throughput 17.16→149.31 tok/s, latency 7.64s→0.48s. Co-LLM (adjusted small model + LLaMA2-70B on GSM8K): throughput 3.46→76.02 tok/s, latency 247.61s→11.46s. R-Stitch: TokenRouter throughput 140.58 tok/s, latency 151.30s; no official baseline for speedup ratio.

System optimization does not mean routing scheme outperforms using only large model on all metrics. E.g., Co-LLM setting: large-model-only throughput 134.79 tok/s > TokenRouter, but end-to-end latency 13.39s > TokenRouter's 11.46s. Throughput, first-token latency, and single-request completion latency must be evaluated separately.

Concurrency scaling 1→16: TokenRouter total throughput grows 8.61× while retaining 51.7% of single-user initial generation speed; official R2R: 5.14× and 31.3%. Paper reports TokenRouter at concurrency 16 vs official R2R at concurrency 1: throughput 18.58×, single-user speed 1.13×. This is cross-concurrency comparison, not 18.58× direct speedup at same concurrency.

Ablation Study

Figure 5 left: concurrency 8 performance breakdown. Right: throughput vs single-user speed trade-off (18.58× and 1.13× from different concurrency points). In R2R at concurrency 8: base implementation 132.78 tok/s, official 134.90. CUDA graph optimization → 230.79 (1.71× official); +async → 296.86; +delayed batching → 372.48 (2.76× official). Gains compose from multiple engineering mechanisms; cannot attribute total improvement solely to routing algorithm.

Different model combos yield different gains: 0.6B+8B: 105.10→336.98 (3.21×); 0.6B+32B: ~2.56×; 1.7B+8B: ~2.76×; 4B+8B: ~1.99×. Model size and speed differences shift bottlenecks; no universal speedup factor.

Appendix notes: increasing large-model tensor parallelism important for throughput; small-model TP increase yields diminishing returns once model fits device. CUDA graph optimization only covers pre-captured batch sizes and token length ranges; outside falls back to regular execution.

Cross-node experiments show feasibility but with communication overhead. Concurrency 4: R2R 228.94→202.43 tok/s (same-node→cross-node); CITER 308.36→277.56. Evidence of deployment feasibility, not proof of covering all WAN or large-cluster challenges.

Advancing the Pareto Frontier

Figure 6: throughput-accuracy trade-off on AMC23. Uses Qwen3-0.6B/32B, greedy decoding, max output 8192, all requests submitted simultaneously, same GPU memory ratio for large/small models. Compares TokenRouter, official R2R, and RouteLLM query-level routing methods (matrix factorization, text encoder, large model, similarity-weighted). Also marks small-model-only and large-model-only reference points.

Runtime optimization makes token-level routing achieve more competitive throughput-accuracy operating points. Algorithms may already have reasonable quality, but high serving cost hindered competition with query-level routing; reducing execution overhead reveals the routing strategy's potential gains. Note: throughput here computed as output tokens per request divided by elapsed time, different from main experiment's aggregate throughput metric. Validates trade-off for specific math task, model combo, and parameter range; does not imply simultaneous speed and accuracy improvement on all tasks, nor that runtime improves model knowledge or reasoning.

Conclusion

TokenRouter's core value: dedicated serving infrastructure for token-level model collaboration. Request-centric algorithm expression, model-centric execution organization; async eliminates global waits, pending-return state reduces re-initialization, delayed batching improves batch utilization.

For algorithm researchers: unified interface separates routing rules from scheduling details, enabling fair comparison of strategies on common runtime. For deployers: model switching cost includes not just extra forward passes but cache management, batch organization, and handoff waits. Evaluating routing schemes requires reporting model combo, concurrency, request length, memory allocation, and precise throughput/latency definitions.

Paper acknowledges boundaries: scheduling model assumes geometric distribution of generation steps between sends; special cases violating this need further study. Theory and experiments do not replace dedicated validation for non-stationary traffic, tail-latency constraints, or larger-scale deployments.

Safest conclusion: TokenRouter proves token-level routing practicality depends not only on "when to switch models" but also on "how to run efficiently after switching". It provides an interface and execution mechanisms that significantly improve serving efficiency across tested algorithms, workloads, and model combos, offering a more usable system foundation for future fine-grained collaboration research.

Original Paper & Resources

Paper: https://arxiv.org/abs/2610.12242 Full text: https://arxiv.org/pdf/2610.12242 Code: https://github.com/thu-nics/TokenRouter This article is based on paper v1. Figures are screenshots from original paper with original text and experimental annotations; captions are Chinese interpretations. Performance numbers apply only to conditions given in the original paper.