Designing a Production‑Ready LLM Gateway: Architecture, Routing, Fallback, and Observability

Problem Analysis

When a system calls only a single model or provider, the workflow is simple: craft a prompt, send an HTTP request, and receive a response. As usage scales, multiple models (e.g., GPT‑4o for complex reasoning, Claude for long‑document analysis, lightweight open‑source models for latency‑sensitive tasks) and providers across clouds are employed. Hard‑coded model names and API keys spread across dozens of micro‑services cause fragility; a sudden OpenAI rate‑limit can bring the entire chain down, and debugging becomes painful.

The LLM Gateway addresses this problem by acting as an API gateway specialized for large‑model calls, centralizing all model invocations and deciding which model to use, how to handle failures, and how to distribute traffic among instances.

Overall Architecture

A production LLM Gateway consists of three layers: ingress, decision, and egress.

Ingress layer receives upstream calls, performs protocol adaptation and parameter normalization, and unifies different SDK formats (e.g., OpenAI vs. Anthropic) into an internal standard. It also handles authentication, rate limiting, and quota checks.

Decision layer is the core brain. After receiving a normalized request, it decides which model to route to, which fallback chain to use if the primary model is unavailable, and which instance should handle the request based on real‑time load and health data.

Egress layer executes the decision: it calls the target provider’s API, processes streaming responses (SSE), converts the response back to a standard format, and records logs and metrics for the entire call chain.

Routing Strategies

Routing is the most technically demanding part because LLM routing is far more complex than traditional API routing.

Capability‑based routing directs requests to the model best suited for the task (e.g., GPT‑4o for complex reasoning, Claude for long‑context tasks, GPT‑4o‑mini for simple classification). The gateway can use task‑type tags or input token length (e.g., >100K tokens → long‑context model).

Cost‑based routing exploits large price differences (up to 10‑100×). A common pattern is “cascade routing”: first try a cheap small model; if its output fails quality checks (confidence score or rule‑based validation), upgrade to a larger model, saving over 50% of call costs.

Latency‑based routing prioritizes the fastest provider for latency‑sensitive requests, using real‑time latency statistics (sliding‑window average or P95) to choose the provider with the lowest current latency.

Semantic routing (Semantic Router) replaces manual rules with an embedding model that vectorizes the request and matches it against predefined task vectors, automatically determining the appropriate model without requiring the caller to specify a task type.

Fallback Mechanism

Fallback is a multi‑layer fault‑tolerance system.

Same‑model retry : retry on the same provider with exponential backoff, limiting retries to 2‑3 and distinguishing retryable errors (timeouts, 429) from non‑retryable ones (400).

Cross‑provider downgrade : if same‑model retries fail, switch to a backup provider, adapting request formats (e.g., OpenAI → Anthropic) transparently.

Model‑level downgrade : if all providers are unavailable or queued, fall back to a lower‑tier model (e.g., GPT‑4o → GPT‑4o‑mini) after business approval of acceptable quality loss.

Bottom‑line strategy : when no model works, return a cached historical answer, a preset default response, or a graceful “service busy” message.

The complete fallback chain is: same‑model retry (with backoff) → cross‑provider switch → downgrade to smaller model → bottom‑line response, each with appropriate timeout thresholds.

A Circuit Breaker monitors consecutive failures per provider; when a threshold is reached, the breaker opens, bypassing the failing provider and routing directly to the fallback chain. After a cooldown, it enters a half‑open state to probe recovery. Because LLM calls naturally have higher latency (seconds to tens of seconds), timeout thresholds are set more loosely than in traditional micro‑services.

Load Balancing in LLM Scenarios

Traditional load balancers (Round Robin, Least Connections) struggle with LLM workloads because request processing times vary dramatically (e.g., 1 s for 50 tokens vs. 30 s for 4000 tokens).

Weighted load balancing : weight considers current concurrent requests, queue depth, and estimated processing time (derived from input token count).

Token‑throughput‑based balancing : distribute load based on total tokens processed per instance rather than request count, keeping token throughput roughly equal across instances.

For self‑hosted models (e.g., vLLM), the gateway can also monitor internal engine metrics such as KV‑Cache usage and batch size to inform decisions.

In multi‑region deployments, proximity routing prefers the nearest instance to reduce network latency, but must be balanced against load (if the nearest instance is saturated, a farther instance may be chosen).

Unified Interface and Protocol Adaptation

A good LLM Gateway exposes a single API that hides provider‑specific protocol differences.

Common practice is to adopt the OpenAI /v1/chat/completions endpoint as the public contract, then internally translate requests to each provider’s native format.

Beyond field name differences, deeper disparities exist: streaming formats (SSE variations), function‑calling payloads, token counting rules, error‑code schemes, and provider‑specific advanced features (e.g., Anthropic’s prompt caching, OpenAI’s structured output). The adaptation layer must normalize these differences or expose optional fields for callers to leverage provider‑specific capabilities.

Cache and Observability

Two indispensable capabilities for a production LLM Gateway are semantic caching and observability.

Semantic cache reduces cost and latency by storing embeddings of past requests in a vector database and returning cached results when similarity exceeds a threshold. GPTCache is a mature open‑source example. Cache freshness and accuracy must be considered; dynamic or creative queries are not suitable for caching.

Observability provides the foundation for operations. The gateway collects metrics such as per‑provider success rate, P50/P95/P99 latency, token consumption, cost, routing rule hit rates, fallback trigger frequency, and failure reasons. These metrics drive alerts and can automatically adjust routing weights (e.g., demote a provider whose latency spikes).

Open‑Source Solutions

LiteLLM : Python implementation supporting 100+ providers, offering a unified OpenAI‑compatible API, built‑in fallback, load balancing, and cost tracking.

RouteLLM : Developed by LMSys, focuses on intelligent routing with a lightweight model that decides whether to use a strong or weak model to save cost while preserving quality.

Portkey AI Gateway : Enterprise‑grade solution with comprehensive observability, caching, retry mechanisms, and fine‑grained control; suitable when production‑level governance is required.