Designing a Production LLM Gateway: Architecture, Routing, and Fallback

Problem Analysis

When a system calls only a single model and provider, integration is trivial: craft a prompt, send an HTTP request, and receive a response. In large‑scale production, multiple models (e.g., GPT‑4o, Claude, lightweight open‑source models) and providers are used simultaneously, with model names and API keys hard‑coded across dozens of micro‑services. A sudden rate‑limit from OpenAI can collapse the entire chain, exposing the need for a centralized LLM Gateway that decides request routing, failure handling, and traffic distribution.

Overall Architecture

A production‑grade LLM Gateway consists of three layers:

Ingress Layer : Receives upstream calls, normalizes protocols and parameters, performs authentication, rate‑limiting, and quota checks. It converts heterogeneous SDK formats (OpenAI, Anthropic, etc.) into an internal standard.

Decision Layer : Core brain that, given a standardized request, determines which model to route to, which fallback chain to use, and which instance should handle the request based on routing rules and real‑time health data.

Egress Layer : Executes the actual API call to the chosen provider, handles streaming responses, converts the provider‑specific response back to the standard format, and logs the full call chain.

Routing Strategies

Routing is the most technically demanding part because LLM calls involve multiple dimensions and trade‑offs.

Capability‑based routing : Directs requests to the model best suited for the task (e.g., GPT‑4o for complex reasoning, Claude for long‑context tasks, GPT‑4o‑mini for simple classification). Task type tags or input token length can trigger routing decisions, such as sending >100K token requests to a long‑context model.

Cost‑aware routing : Implements "cascade routing"—first try a cheap model, evaluate output quality via confidence scores or rule checks, and upgrade to a larger model only if needed. This can reduce costs by more than 50%.

Latency‑aware routing : Maintains real‑time latency statistics (e.g., sliding‑window average or P95) for each provider and prefers the fastest provider for latency‑sensitive requests.

Semantic routing : Uses a lightweight embedding model to vectorize the request and match it against predefined task vectors, automatically determining the appropriate model without explicit task tags.

Fallback Mechanism

Fallback is a multi‑layered fault‑tolerance system rather than a simple retry.

Same‑model retry : Retry 2‑3 times on the same provider with exponential backoff; only retry on timeout or 429 rate‑limit, not on 400 parameter errors.

Cross‑provider downgrade : If retries fail, switch to a predefined backup provider, adapting request formats (e.g., OpenAI → Anthropic) transparently.

Cross‑model level downgrade : If high‑end models are unavailable, fall back to a lower‑tier model (e.g., GPT‑4o → GPT‑4o‑mini) after business approval.

Catch‑all strategy : When all models are unavailable, return cached historical results, a preset default answer, or a graceful "service busy" message.

The complete fallback chain is: same‑model retry (with backoff) → cross‑provider switch → downgrade to smaller model → final catch‑all.

A circuit breaker monitors consecutive failures per provider; once a threshold is reached, the provider is temporarily bypassed, and after a cool‑down period it enters a "half‑open" state to probe recovery. Timeout thresholds are looser than in typical micro‑services because LLM calls naturally take seconds.

Load Balancing in LLM Scenarios

Traditional load balancers (Round Robin, Least Connections) struggle because LLM request latencies vary widely (e.g., 1 s for 50 tokens vs. 30 s for 4000 tokens).

Weighted load balancing : Weights consider current concurrent requests, queue depth, and estimated processing time based on input token count.

Token‑throughput‑based balancing : Distributes load based on total tokens processed per instance rather than request count.

For self‑hosted models (e.g., vLLM), the gateway can also monitor internal engine metrics such as KV‑Cache usage and batch size to inform decisions.

Proximity routing in multi‑region deployments prefers the nearest instance but falls back to a less‑loaded distant instance when necessary.

Unified Interface and Protocol Adaptation

The gateway should expose a single API (commonly the OpenAI /v1/chat/completions endpoint) while internally translating to each provider's native format.

Beyond field name differences, providers differ in streaming implementations, function‑calling payloads, token counting, error codes, and exclusive features (e.g., Anthropic's prompt caching, OpenAI's structured output). The gateway must normalize these differences or expose optional extensions for provider‑specific capabilities.

Cache and Observability

Semantic cache reduces cost and latency by embedding incoming requests and performing similarity search in a vector database; a hit returns the cached result. GPTCache is a mature open‑source implementation, though it may suffer from staleness or inapplicability to creative queries.

Observability is essential for operations. The gateway collects metrics such as provider success rates, P50/P95/P99 latency, token consumption, cost, routing rule hit rates, fallback trigger frequencies, and reasons. These metrics drive alerts and can automatically adjust routing weights (e.g., lowering weight for a provider whose latency degrades).

Open‑Source Solutions

Common open‑source LLM Gateway projects include:

LiteLLM : Python implementation supporting 100+ providers, offers OpenAI‑compatible API, built‑in fallback, load balancing, and cost tracking.

RouteLLM : Developed by LMSys, focuses on intelligent routing via a lightweight routing model that decides between strong and weak models to save cost while preserving quality.

Portkey : Enterprise‑grade gateway with comprehensive observability, caching, retry, and other controls.

Selection depends on team needs: quick multi‑provider unification (LiteLLM), cost‑optimizing routing (RouteLLM), or production‑grade observability and governance (Portkey).

Reference Answer

The LLM Gateway is essentially an API gateway for large‑model calls, composed of ingress, decision, and egress layers. The ingress normalizes protocols and enforces auth/ratelimiting; the decision layer handles routing, fallback orchestration, and load distribution; the egress performs the actual calls and response conversion.

In practice, routing combines capability‑based routing, cost‑aware cascade routing, latency‑aware dynamic selection, and optional semantic routing via embeddings.

Fallback is a four‑stage waterfall: same‑model retry with exponential backoff, cross‑provider switch with format adaptation, downgrade to a smaller model, and finally a catch‑all strategy. A circuit breaker skips repeatedly failing providers.

Load balancing must account for the huge variance in request processing time; weighted balancing based on concurrency, queue depth, and token throughput is required. The implementation often builds on LiteLLM, adds semantic caching, and a full metrics suite to continuously refine routing.