When to Deploy an LLM Gateway? Complete Guide to Architecture, Routing, Cost, and Security

When a project starts with a single model SDK, the code is simple, but as multiple models and teams are added the system quickly suffers from fragmented interfaces, invisible costs, scattered API keys, lack of unified rate limiting, and poor observability. The article argues that these issues are not merely "model‑related" but stem from trying to manage multi‑model complexity without a governance layer.

Why a Gateway Matters

The gateway does more than provide a uniform API; it centralises model access, enables intelligent routing (by cost, latency, or business priority), provides automatic failover, aggregates token usage for budgeting, enforces virtual keys for security, and offers a single point for logging and tracing.

When to Adopt a Gateway

Multiple models or providers are in use.

Several business lines share model capabilities.

Cost, quota, or permission management is required.

Unified logging, tracing, and audit are needed.

Model switching should be a configuration change, not code change.

Gateway Options and Their Fit

LiteLLM Proxy – OpenAI‑compatible, supports 100+ models, ideal for teams comfortable with OpenAI‑style SDKs and willing to self‑host.

One API / New API – Best for Chinese teams using domestic models, offers channel management and a graphical UI.

OpenRouter – Hosted, low‑cost entry point for rapid experimentation.

LangChain – Application‑level abstraction; good for elegant model switching inside code but not a full governance platform.

Portkey Gateway – Enterprise‑grade with workflow, compliance, and advanced governance.

Kong AI Gateway – Plugin for organisations already using Kong API gateway.

Getting Started with LiteLLM

A minimal deployment can be launched with the following docker‑compose.yml:

version: "3.8"
services:
  litellm:
    image: ghcr.io/berriai/litellm:main-latest
    ports:
      - "4000:4000"
    volumes:
      - ./litellm_config.yaml:/app/config.yaml
    environment:
      - DATABASE_URL=postgresql://user:pass@postgres:5432/litellm
      - LITELLM_MASTER_KEY=sk-your-master-key-change-this
    depends_on:
      - postgres
  postgres:
    image: postgres:15
    environment:
      - POSTGRES_DB=litellm
      - POSTGRES_USER=user
      - POSTGRES_PASSWORD=pass
    volumes:
      - postgres_data:/var/lib/postgresql/data
volumes:
  postgres_data:

The core configuration lives in litellm_config.yaml. Important sections include:

model_list – defines each model, its provider SDK name, API key, and rate limits.

router_settings – chooses a routing strategy (e.g., "usage‑based" or "latency‑based"), enables load‑balancing, sets retry counts, and defines fallbacks.

general_settings – master key, database URL, and whether to store models in the DB.

Key questions to resolve in the config are which models are primary vs. fallback, routing priorities (cost vs. stability), retry limits, and which models are exposed to which applications.

Virtual Keys and Cost Governance

Instead of handing raw provider keys to services, generate virtual keys via the gateway:

curl http://localhost:4000/key/generate \
  -H "Authorization: Bearer sk-your-master-key-change-this" \
  -H "Content-Type: application/json" \
  -d '{
    "models": ["gpt-4o", "deepseek-chat"],
    "max_budget": 50.0,
    "budget_duration": "1mo",
    "rpm_limit": 100,
    "tpm_limit": 10000,
    "metadata": {"user": "[email protected]", "team": "marketing"}
  }'

This makes it easy to see which app uses which model, enforce per‑user or per‑team budgets, and block requests that exceed limits.

Routing Strategies

Two built‑in strategies are demonstrated:

Cost‑first routing – directs cheap tasks to low‑cost models and only uses expensive models for high‑value requests.

Latency‑first routing – prefers the fastest model for real‑time use cases.

For fine‑grained business logic, a custom router can be written in Python:

import litellm
from litellm.integrations.custom_logger import CustomLogger

class CustomRouter(CustomLogger):
    async def async_pre_call_hook(self, user_api_key_dict, cache, data, call_type):
        user_query = data.get("messages", [])[-1].get("content", "")
        if "紧急" in user_query:
            data["model"] = "gpt-4o"
        elif "总结" in user_query or "翻译" in user_query:
            data["model"] = "deepseek-chat"
        else:
            data["model"] = "claude-sonnet-4"
        return data

litellm.callbacks = [CustomRouter()]

The principle is that routing decisions should reflect business value, not just model parameters.

Cost Tracking and Alerts

Define per‑token costs in the model definition and query spend via the gateway:

model_list:
  - model_name: gpt-4o
    litellm_params:
      model: openai/gpt-4o
      api_key: ${OPENAI_API_KEY}
      input_cost_per_token: 0.000005
      output_cost_per_token: 0.000015

Spend can be inspected with:

curl http://localhost:4000/spend/keys?api_key=sk-xxx \
  -H "Authorization: Bearer sk-your-master-key"

Alerting thresholds (e.g., 80% of budget) can be sent to Slack via webhook configuration.

Observability

Integrate callbacks such as Langfuse to capture success and failure traces:

general_settings:
  success_callback: ["langfuse"]
  failure_callback: ["langfuse"]

litellm_settings:
  langfuse_public_key: ${LANGFUSE_PUBLIC_KEY}
  langfuse_secret_key: ${LANGFUSE_SECRET_KEY}
  langfuse_host: "https://cloud.langfuse.com"

With observability in place, teams can answer questions like which model fails most, where latency spikes originate, and which requests trigger fallbacks.

Production‑Ready Architecture

In production, a single gateway instance becomes a single point of failure. The recommended topology includes:

External load balancer (e.g., Nginx) distributing traffic to multiple LiteLLM instances.

Shared PostgreSQL (primary/replica) and Redis for state, quotas, and hot configuration.

Backend provider services (OpenAI, Anthropic, DeepSeek) behind the gateway.

Key considerations are multi‑instance deployment, shared database for consistent budgets and audit logs, Redis for fast rate‑limit checks, and load‑balancer for high availability.

Security Best Practices

Never expose raw provider keys to downstream services; use virtual keys generated by the gateway.

Apply least‑privilege scopes to each virtual key (specific models, budgets, IP allow‑lists).

Rotate underlying provider keys centrally without touching applications.

Enable detailed audit logs (store_model_in_db=true) to trace who called which model, cost, and errors.

Place content‑safety checks at the gateway entry point to enforce compliance before downstream processing.

Selection Matrix (Scenario → Recommended Solution)

Personal project / quick experiment → OpenRouter (hosted, low friction).

Startup with mixed models → LiteLLM Proxy (lightweight, OpenAI‑compatible).

Domestic‑model‑centric team → One API / New API (good UI, channel management).

Enterprise production with HA → LiteLLM Proxy cluster (self‑hosted, scalable).

Existing Kong infrastructure → Kong AI Gateway plugin.

Deep LangChain integration → LangChain + LangServe (application‑level abstraction).

Need strong compliance and workflow → Portkey Gateway (enterprise features).

Implementation Roadmap

Define governance boundaries: is the goal unified access or full platform governance?

Select a primary solution based on scenario and team capability.

Deploy a minimal gateway with 2‑3 key models.

Configure virtual keys and budget limits before scaling.

Integrate monitoring and tracing (e.g., Langfuse, Prometheus).

Iteratively add fallback models and fine‑tune routing based on real traffic.

When traffic grows, move to a multi‑instance, load‑balanced deployment.

Key Takeaways

The essence of an LLM gateway is governance, not merely request forwarding.

Adopt a gateway once model usage becomes a platform‑wide concern.

Choose a solution that matches existing infrastructure before adding features.

Virtual keys, routing, budgeting, and observability deliver real value.

Without proper permission and cost controls, the gateway merely relocates complexity.

High availability and shared state design prevent the gateway itself from becoming a bottleneck.