LiteLLM: A Unified Gateway for Over 100 LLM APIs

Problem: API dialect friction

Switching between LLM providers (e.g., OpenAI GPT‑4, Anthropic Claude, Azure OpenAI, local VLLM) requires rewriting request code, adapting parameter formats, and handling different error responses.

Unified gateway – LiteLLM

LiteLLM acts as an AI gateway that exposes a single, OpenAI‑compatible interface for more than 100 LLM APIs, including OpenAI’s full suite, Anthropic Claude, Google Vertex AI, AWS Bedrock, Azure OpenAI, Cohere, Groq, Hugging Face inference endpoints, local VLLM, and NVIDIA NIM.

Usage modes

Python SDK – lightweight integration for existing Python code.

Proxy Server (AI Gateway) – deployable as an independent service providing enterprise‑grade features.

Proxy Server capabilities

Cost tracking & budget control – records token usage and expenses per user/project; supports budget limits.

Load balancing & failover – distributes requests across multiple instances of the same model (e.g., several GPT‑4 keys) and switches to a backup model on failure.

Rate limiting & authentication – manages per‑client request rates and protects real API keys via virtual keys.

Detailed logging & audit – logs every request and response for debugging, analysis, and compliance.

Quick start (≈5 minutes)

Install the SDK: pip install litellm Set API keys as environment variables and invoke the unified completion function. Changing only the model prefix switches providers while all other parameters stay identical.

from litellm import completion
import os
os.environ["OPENAI_API_KEY"] = "your-key"
os.environ["ANTHROPIC_API_KEY"] = "your-key"
# OpenAI GPT‑4o
response = completion(model="openai/gpt-4o", messages=[{"role": "user", "content": "Hello!"}])
# Anthropic Claude Sonnet
response = completion(model="anthropic/claude-sonnet-4-20250514", messages=[{"role": "user", "content": "Hello!"}])

The only required change is the model prefix (e.g., openai/ or anthropic/); the messages payload format remains unchanged.

Typical adopters

Researchers and developers conducting multi‑model experiments who need rapid prompt testing across state‑of‑the‑art models.

Start‑ups seeking stability and cost efficiency via load balancing, failover, and fine‑grained budgeting.

Enterprises building internal AI platforms that require a unified access point, centralized permission, quota, and expense management.

Builders of complex AI applications (e.g., autonomous agents) that leverage LiteLLM’s A2A protocol support and integration with frameworks such as LangGraph and Pydantic AI.