Two-Level AI Inference Gateway: Prefix-Aware Routing & vLLM Router Fixes

Background: Inference Service Deployment and Routing Challenges

Inference services run in Kubernetes. The same model may have multiple deployment instances across clusters (cross-cluster) or within a cluster (different machine types, networks, deployment shapes). Each instance contains multiple Pod replicas sharing a KV Cache pool. When a user requests a model, the gateway must make two choices: first select an instance, then select a replica within that instance.

Unlike ordinary HTTP gateways that assume uniform request cost, short latency, and stateless backends, inference requests differ in three key ways:

Unequal request cost: Context length varies widely; two backends with 10 in-flight requests each can differ in actual load by an order of magnitude. Round-robin or random distribution is unsuitable.

Cache-state affinity required: Inference engines keep KV cache for processed prefixes. Switching backends mid-conversation forces a full prefill redo, multiplying first-token latency. The "least-loaded" backend is often not the fastest.

Heterogeneous capabilities hard to codify: Different hardware yields different throughput, latency, and concurrency limits for the same model. Static weights reflect only provisioning-time state and drift from reality.

Two-Level Routing Architecture Overview

The architecture uses two routing levels:

Level 1 (Global): Higress + custom WASM plugin aitc-ai-lb — selects target inference instance. Deployed once globally.

Level 2 (Per Instance): Self-maintained vllm-router — selects target replica within the chosen instance. One deployed per inference instance.

Level 1 picks an instance based on real-time latency, in-flight requests, health, and request prefix. Level 2 picks a replica inside the chosen instance.

Level 1 Gateway: Higress + aitc-ai-lb

Higress Role

Higress is an AI-native API gateway built on Istio and Envoy, validated at Alibaba scale. It provides:

Unified entry & routing: Routes by domain, path, model name (e.g., GLM-5.3) to the corresponding AI routing rule, then hands off to aitc-ai-lb for instance selection.

Authentication: Consumer auth and security governance.

Observability foundation: Access logs, Envoy metrics, traffic metrics.

WASM extensibility: Hot-updatable custom plugins; instance-level inference routing is implemented via the custom aitc-ai-lb WASM plugin.

Higress's built-in load balancing and official AI routing plugin do not cover cross-cluster heterogeneous instances, multi-signal scoring, or prefix-affinity routing, necessitating aitc-ai-lb.

aitc-ai-lb Core Capabilities

The plugin handles the full loop: candidate collection, state awareness, routing decision, request rewrite, and result observation. Its routing target is a deployment instance, not individual workers.

Prefix matching: Matches request path prefixes to target services, enabling independent routing rules per model, business interface, or tenant.

Routing scoring: Dynamically scores candidates using backend health, real-time load, resource utilization, historical response latency (TTFT/E2E EWMA), concurrency, etc., preferring higher scores.

Affinity/anti-affinity hard filters: Binds request features (e.g., HTTP headers) to backend services for tenant isolation, A/B testing, multi-active/DR routing.

Fallback service: Auto-forwards to a preset fallback when no candidate matches or all are unavailable.

Monitoring metrics: Exposes Prometheus metrics for request volume, success/error rates, latency, routing hits, backend health, instance load, fallback traffic.

Fault isolation & recovery: Auto-ejects unhealthy instances, periodic health probes, auto-rejoin on recovery.

Routing Decision Pipeline

Each request passes through a defined sequence:

Collect candidates from configuration.

Health filter removes unhealthy services.

Affinity filter applies configured affinity/anti-affinity rules.

Prefix match — if a prefix-matched backend exists, all others are dropped.

Load metrics load — reads TTFT/E2E EWMA, running, pending for each candidate.

Admission rules — validates candidates against thresholds; those passing proceed to multi-signal scoring , others are dropped; if none pass, go to fallback or reject .

Multi-signal scoring — min-max normalizes each metric, applies weights, multiplies by service weight.

Selection & hit threshold — chooses backend via global_best (highest score) or p2c (power of two choices: random pair, pick better) to mitigate thundering herd.

Fallback or reject — if fallback configured, retry there; if overload-reject configured, return 503/429.

Complete & forward — forwards request, records metrics.

The overall philosophy: hard constraints narrow the candidate pool, real-time state drives soft scoring, prefix hits prioritize KV cache affinity, layered fallback handles failure domains .

Model Mapping & Unified Entry

aitc-ai-lb

exposes a unified model name to clients while allowing multiple backend services under the same instance to use different upstream model names. On forward, the plugin rewrites the client's model to the target service's required model. A single Higress route can onboard services with different model names and deployment shapes; callers need not know backend naming differences.

Multi-Signal Intelligent Routing & Overload Protection

After health and hard-filter passes, ordinary requests enter an ordered policy chain. Each policy contains four steps:

Admission: Thresholds on TTFT, E2E, running, pending. Notably, if a backend's running count is below prefer_serving_below, it is marked a "serving backend" and gets priority even if historical TTFT is high or exceeds admission thresholds — preventing starvation of idle backends.

Rate limit / serving pre-filter: Limits per-service running-request share within the gateway pod; when low-load serving backends exist, routing prefers that subset.

Multi-signal scoring: Min-max normalization per metric, weighted sum, multiplied by service weight.

Selection & hit threshold: global_best or p2c.

To reduce thundering herd, aitc-ai-lb stores running/pending metrics in Envoy SharedData so subsequent requests in the same gateway pod see fresh load instantly.

Prefix Cache Mechanism

When enabled:

Request prefix–backend bindings saved to Redis.

On prefix hit, the matched backend becomes the sole candidate.

Admission rules are relaxed for cache hits.

Relaxation rationale:

Cache hits skip full prefill; marginal cost is decode only. The backend can handle more requests at same load, so thresholds (e.g., ttft_ms) can be loosened 2–3×. Diverting a lightly overloaded cache-hit backend to a cold backend for full prefill is more expensive.

Relaxation has limits: if load exceeds the relaxed admission rule, the request spills over to avoid hammering a single backend.

Prefix cache expiry uses Redis TTL (refreshed on hit). If a cache-hit backend returns an abnormal response, the plugin auto-evicts that prefix binding to prevent caching errors.

Request Feature Affinity Routing

Supports hard filtering based on request features (HTTP headers). Use cases:

Tenant isolation — dedicated instances per tenant.

Application/business traffic isolation — separate services per app/module.

A/B testing or canary — route by user ID or version tag.

Multi-active/DR — route by user region to nearest/designated data center.

If filtering yields zero candidates, fallback is preferred over hard rejection to preserve availability.

Health Monitoring & Self-Healing

Repeated 4xx/5xx responses accumulate failures; health manager can eject the backend.

Envoy tick mechanism periodically probes unavailable backends; consecutive successes restore health.

Fault Tolerance & Overload Protection

Fallback (fail-open default): When primary pool has no healthy backend, selector empty, Redis read fails, or policy chain misses, the plugin falls back by severity: switch to configured fallback_services; if none, weighted round-robin across preset services.

Overload reject (fail-fast exception): Only when all candidates exceed hard thresholds and metrics are trusted does it fast-fail with 429 (503 not observed in code). Infrastructure anomalies (missing health, Redis down) always fail-open to prevent control-plane/storage faults from cascading into ingress outage.

Observability

Routing logs: Each decision logs influencing policy, target service, upstream model, fallback trigger; prefix hits log backend and prefix hash; fallbacks log candidate metric snapshots for post-mortem.

Metrics: Prometheus metrics covering running/pending, TTFT, E2E, status codes, backend health, prefix hit/overflow/eviction, Redis latency.

Level 2 Gateway: Official Limits & Custom Strategies

Level 2 uses the official vllm-router — a lightweight router per instance that discovers worker Pods via K8s label selector and distributes requests. Chosen because most models run on vLLM, so vllm-router reuses the ecosystem (service discovery, health checks, Prefill/Decode disaggregation scheduling). However, the upstream version evolves slowly and its load accounting and routing strategy proved unreliable in production.

Shortcomings of Upstream vllm-router

Strategy mismatch: The load-aware strategy power_of_two randomly picks two workers and sends to the less loaded. With few workers, scanning all loads is cheap; random sampling wastes the opportunity to avoid already-overloaded workers.

Unreliable load metrics: power_of_two relies on worker-reported /get_load, which some deployment shapes cannot provide. Under Prefill/Decode disaggregation, reported values diverge from actual in-flight requests. The router's own counter is worse: manual +1/-1 pairing across 20+ call sites (scattered in error branches, streaming closures). Any early-return, client disconnect, or task cancellation that misses a -1 permanently inflates that worker's count — it stays at the bottom of the queue forever, receiving no traffic until process restart. The leak is invisible externally; the worker just looks perpetually idle.

Custom Improvements Summary

New strategy soft_k_choice — addresses shortcoming 1 ( power_of_two random pair admits overloaded workers): takes globally lowest-loaded k workers, then weighted-random by in-flight among them; k=1 degrades to strict least-loaded.

In-flight counting rebuild — addresses shortcoming 2 (manual +1/-1 leaks cause permanent inflation): RAII guard auto-decrements on destruction; time-bucket expiry prunes leaks, max residue one TTL.

Both changes together yield "trustworthy load + dispersed routing": without trustworthy in-flight counts, any load-aware strategy is meaningless.

soft_k_choice Strategy Detail

Router maintains exact in-flight counts as load. On routing, it takes the k globally least-loaded workers, then picks one via weighted random (lower load = higher weight). Added randomness spreads concurrent requests across multiple workers instead of always picking the single best, dissolving thundering herd at the final hop.

Key differences from power_of_two:

Candidate set: Global lowest-loaded k, not a random sample of two — overloaded workers never enter the candidate set.

Final hop: Weighted random, not deterministic pick. Concurrent requests see the same top- k snapshot but choose randomly, naturally scattering across the k workers.

Default k=2; k=1 = strict least-loaded. Prefill and Decode can each specify their own strategy. Prometheus metric vllm_router_running_requests tracks this strategy's in-flight, making load changes externally visible.

Precise In-Flight Counting: RAII Guard + Time-Bucket Self-Healing

Two layers ensure correctness:

Layer 1: RAII Guard Eliminates Missed Decrements

Counting no longer relies on manual +1/-1 callbacks. When a request occupies a worker, it acquires a guard; guard destruction auto-decrements. Non-streaming requests destroy on function return; streaming requests move the guard into the forwarding closure — normal completion, mid-stream error, client disconnect all trigger destruction. The 20+ manual call sites on the Prefill/Decode path are removed; each phase holds its own guard; retry loops acquire a fresh guard per attempt, so no single retry can leak. Correctness is guaranteed by language destructors, not human memory.

Layer 2: Time-Bucket Fallback Self-Healing

Guards could still leak (e.g., accidentally placed in a never-ending task). Therefore the counter is not a single integer but bucketed by time: each worker's in-flight is recorded in fixed-width buckets (default 5 seconds). Acquire logs to current bucket; release subtracts from the exact originating bucket. Buckets older than TTL (default 600 seconds) are pruned wholesale — any leaked count in that bucket zeroes automatically, capping residue to at most one TTL without process restart. Routing reads the aggregate of all live buckets; lazy pruning keeps hot-path overhead negligible. Monotonic clock avoids NTP rollback falsely expiring all buckets.

Observability Hooks

Pruning logs a warning with worker address and pruned count if the bucket's count is non-zero. Normally buckets should be empty at expiry; non-zero means either a genuine leak or a request that ran the full TTL — the only evidence a leak ever occurred, since everything else looks normal after self-healing.

Prometheus gauge re-emits after pruning; otherwise the metric would stall at the leaked high value.

Effect Comparison

Tests comparing power_of_two vs soft_k_choice under high traffic show worker request counts. soft_k_choice yields markedly more balanced load across workers.

Summary

This two-level inference gateway has run stably in production. Core value:

Reduced system complexity: Level 1 handles cross-cluster instance selection; Level 2 handles intra-instance replica selection. Clear responsibility boundaries split a complex problem, lowering development and configuration overhead, enabling independent releases and evolution per layer.

Handles high traffic: Stably processes million-request-per-day volumes, accommodating long-running streaming inference requests.

Routing strategies tailored to inference: Level 1 combines model constraints, real-time load, prefix affinity, and overload state for instance selection; Level 2 filters low-load candidates then uses weighted random to disperse concurrent requests. Together they improve KV cache utilization while preventing traffic concentration on a few instances or replicas, achieving more balanced load distribution under high throughput.