Inference Frameworks Demystified: Core Concepts, Frameworks & Cluster Deployment for LLMs

1. Model Lifecycle and the Role of Inference Frameworks

A large model goes through four stages: pre-training, post-training (SFT, RL, distillation), evaluation, and deployment. Inference refers to using a trained model to compute outputs from inputs. An inference framework loads models onto hardware, receives requests, organizes computation, and returns results. It must balance five goals: fast (low first-token latency, smooth generation), high throughput (serve many requests concurrently), cost-efficient (maximize GPU memory and compute reuse), stable (handle traffic spikes, long requests, hardware failures), and correct (output matches model capability, format compliance, reproducibility). These goals often conflict; the framework finds the right trade-off for the business.

Inference frameworks are used not only at serving time but also during post-training (RL needs massive generation), evaluation (running thousands of tests), and synthetic data generation. Unlike training (one-time cost), inference is a recurring expense that grows with usage.

2. Why Agents Demand More from Inference Frameworks

Agents execute multi-step tasks with tool calls, causing long, repetitive contexts. Each step re-sends system prompts, tool definitions, and prior history. This creates three characteristics: long inputs, short outputs, high inter-request repetition. End-to-end latency accumulates across steps. Frameworks must now optimize not just raw speed but also reuse of already-computed work (caching), directly impacting cost and latency.

3. Training vs. Inference Frameworks

Key differences:

Core Goal : Training frameworks train the model; inference frameworks serve the model.

Computation : Training does forward + backward, updates weights; inference does forward only, weights fixed.

Load Pattern : Training uses batched, planned loads; inference faces random arrival, variable length.

Time Sensitivity : Training accepts days/weeks; inference requires seconds or less per request.

Memory Usage : Training stores weights, gradients, optimizer states, activations; inference stores weights and KV Cache.

Key Metrics : Training focuses on throughput, convergence, utilization; inference focuses on TTFT, TBT, throughput, unit cost.

Inference frameworks resemble online service systems (databases, web servers) but schedule GPUs and VRAM instead of CPUs and disks.

4. Terminology Clarification

Inference Engine : Core program running model on one/few machines: scheduling, memory, compute. Typical form: server process or Python library.

Inference Framework : Often used interchangeably; broader: includes API, distributed deployment, caching. Typical form: open-source software suite.

Inference Service : Deployed framework exposing online service with gateway, auth, billing, monitoring, autoscaling. Typical form: enterprise model cluster.

MaaS Platform : Cloud vendor product; pay-per-call, no infra management. Typical form: cloud provider model APIs.

Local Runner : Bundles engine, model download, simple API for personal devices. Typical form: desktop tool (one command).

5. Core Concepts

Token

Smallest unit models read/write. English: ~0.75 words/token; Chinese varies. Billing unit for APIs; output tokens cost several times more than input tokens.

Prefill and Decode

Prefill : processes all input tokens in parallel (compute-bound). Produces first output token. Decode : generates tokens sequentially; each step reads entire model weights from VRAM but does little compute (memory-bandwidth-bound). This explains why output is pricier: serial generation occupies GPU longer per token.

Continuous Batching

Early batching waited for all requests to finish. Continuous batching schedules per decode step: finished requests exit, new ones enter immediately. Now standard in mainstream engines.

KV Cache

Attention requires all prior tokens. Recomputing each step is quadratic. Solution: store Key/Value projections per layer per token — the KV Cache. Acts as model's scratchpad. Size grows with context length and concurrency; often the tightest VRAM resource. Quantization (FP16→INT8/INT4) and new attention architectures (e.g., MLA) compress it. Model fit depends on weight VRAM; concurrency depends on KV Cache VRAM.

Performance Metrics: TTFT, TBT, Throughput

TTFT (Time To First Token): latency to first output, driven by Prefill.

TBT (Time Between Tokens) / TPOT: average inter-token interval, driven by Decode.

Throughput: tokens or requests per second; determines hardware efficiency and unit cost.

These conflict: larger batches raise throughput but increase per-request latency. Frameworks maximize Goodput — throughput meeting latency SLOs. For agents, total task time = sum of per-step TTFT + generation time + tool latency; small per-step delays compound.

Mixture of Experts (MoE)

Models like DeepSeek-V3 (671B total, 37B active per token) use sparse activation. All experts must reside in VRAM, often requiring multi-GPU placement. Cross-GPU communication and expert load balancing become new framework challenges.

6. Landscape of Open-Source Inference Frameworks (as of Oct 2026)

General-Purpose Server Engines

vLLM (UC Berkeley SkyLab, Jun 2023): PagedAttention for KV Cache paging; PyTorch Foundation project (May 2025). ~94k GitHub stars.

SGLang (LMSYS, Jan 2024): RadixAttention for automatic prefix caching; strong structured generation. ~37k stars.

TensorRT-LLM (NVIDIA, Oct 2023): Deep optimization for NVIDIA GPUs. ~15k stars.

LMDeploy (Shanghai AI Lab / InternLM, 2023): Integrates compression, deployment, serving. ~8k stars.

Hugging Face TGI (2022) entered maintenance mode Dec 2025, recommending migration to vLLM/SGLang.

Two trends: (1) research-to-production lag is tiny — continuous batching (Orca 2022), PagedAttention (2023), FlashAttention (2022), speculative decoding (2022-23), PD separation (DistServe/Splitwise 2024) all adopted quickly. (2) Feature convergence: continuous batching, paged KV Cache, prefix caching, chunked prefill, speculative decoding, structured output are now table stakes. Competition shifts to new-model support speed, hardware coverage, and large-cluster behavior.

Local & Edge Runners

llama.cpp (Georgi Gerganov, Mar 2023): C/C++, runs on CPUs/Apple Silicon. GGUF format (Aug 2023) became standard for quantized local models. ~130k stars.

Ollama (2023): wraps download/run/API in one command. Initially used llama.cpp; since 2025 moving to custom GGML-based engine. ~180k stars (highest in inference).

MLC-LLM : ML compilation for phones, browsers.

Higher stars reflect larger hobbyist/developer audience vs. cluster engineers.

Cluster Orchestration Layer (2025+)

New projects sit above engines, managing whole clusters: request routing, PD separation, cross-machine KV Cache transfer, autoscaling.

NVIDIA Dynamo (GTC Mar 2025): PD separation, KV-aware routing, KV management/transfer; works with TensorRT-LLM, vLLM, SGLang.

llm-d (May 2025, Red Hat + CoreWeave, Google Cloud, IBM, NVIDIA): Kubernetes-native distributed inference built on vLLM.

Mooncake (Moonshot/Kimi): KV-centric PD separation; open-sourced data transport, KV storage; FAST 2025 best paper.

Drivers: models too large for single machine; agent workloads make cross-request/machine KV reuse highly valuable. As single-node optimizations converge, cluster scheduling becomes the differentiator.

Model Vendor & Internet Company Systems

DeepSeek : Open Source Week Feb 2025 released FlashMLA, DeepEP, DeepGEMM, EPLB, 3FS; published V3/R1 system design. Internal engine forked from early vLLM, tightly coupled to infra; chose to contribute components and pre-sync with community for day-zero support. DeepInfer open-sourced Oct 2026.

Moonshot Mooncake (above).

Alibaba RTP-LLM .

Paths differ (components vs. full system), but consensus: as model complexity and deployment scale grow, inference implementation becomes part of model capability.

Why New Models Need Framework Adaptation

Frameworks don't auto-run arbitrary models. Each new model requires: implementing novel architecture (attention, expert routing), writing fast kernels, aligning chat templates/tool formats, supporting new quantization formats, multimodal inputs. Examples: DeepSeek V3.2-Exp (sparse attention) and Qwen3-Next (hybrid attention) both had vLLM support on release day thanks to pre-release vendor-framework collaboration — becoming industry norm.

Hardware Diversification

Beyond NVIDIA CUDA: vLLM 2025 plugin mechanism enables external hardware backends (vllm-ascend for Huawei Ascend, vLLM-Kunlun for Baidu Kunlunxin, Moorse for Iluvatar). JD's xLLM focuses on domestic chips; DeepInfer supports NVIDIA + Ascend. Model designs now consider domestic chips (DeepSeek V3.1 UE8M0 FP8 for next-gen domestic chips). Triton, TileLang raise kernel portability. More options but validation still needed per chip/framework combo. More hardware diversity increases framework value as the adaptation layer.

7. Three Evolutionary Directions

Single-node engine → cluster system

Generic implementation → co-design with models

Single hardware → multi-hardware support

Driven by LLMs becoming large-scale, continuously running infrastructure where inference efficiency determines viability.

8. Request Journey Through the Framework

Example: 8k-token request to enterprise customer-service agent (system prompt, tools, history + new query).

1. Entry: Protocol → Token Sequence

Normalize protocols (OpenAI, etc.) to model-expected format. Core: chat template — concatenates system, user, assistant, tool calls/results with special tokens per model's training format. Tokenizer converts to IDs. Template mismatch degrades quality and breaks tool calls. Security: special tokens are control instructions; user/tool content containing them must be escaped to prevent injection. Response: detokenize, parse thinking/content/tool calls, stream back per protocol. This layer determines consistency across deployments.

2. Scheduling: Batching Decisions

Scheduler is the command center. Per step: which requests to process? Start new Prefill? Continue Decode? Enough KV Cache space? Implements continuous batching. For long requests: Chunked Prefill splits Prefill into chunks interleaved with Decode of other requests, avoiding head-of-line blocking. Manages VRAM admission: queue or evict (swap to CPU, drop) when KV Cache full. Async scheduling overlaps CPU prep with GPU compute to eliminate bubbles. Multi-machine: cluster load balancer routes to instance (considering queue length, VRAM, prefix cache locality), then instance scheduler handles per-step decisions. Alibaba Cloud PAI-EAS LLM Smart Routing offers prefix-cache, least-requests, least-tokens strategies.

3. Execution: Parallelism & Kernels

When model exceeds one GPU: Tensor Parallel (TP) splits per-layer matrices; Pipeline Parallel (PP) splits layers across GPUs; Data Parallel (DP) replicates model for different requests; Expert Parallel (EP) places different MoE experts on different GPUs. Large MoE often combines DP+EP (dozens/hundreds of GPUs). Hot experts replicated for load balance. Kernels (matrix mul, attention, expert routing) determine speed; FlashAttention reduces VRAM traffic. Low-precision (INT8/INT4) saves VRAM and uses tensor cores. CUDA graphs record/replay kernel launches to cut launch overhead in Decode. Model structure defines what to compute; kernels define how fast.

4. Caching: Reuse Computed Work

System prompt, tool defs, prior turns are identical across requests. Prefix Caching reuses their KV Cache. Early frameworks reserved contiguous VRAM per request (wasted). PagedAttention (vLLM) pages KV Cache into fixed-size blocks (OS-like), enabling sharing and reducing fragmentation. Multi-level cache: hot in VRAM, warm in CPU RAM, cold in SSD/distributed store (e.g., Alibaba Cloud Tair KVCache pools VRAM/RAM/storage, adapts vLLM/SGLang/TRT-LLM). Cluster router prefers instances with needed prefix cache. Cache hit = skip Prefill = huge cost drop ; providers discount cached input tokens to 1-2% of original. Agent dev tip: keep request prefix stable (fixed system prompt/tools first, variable content last); prepending timestamps invalidates all downstream cache. In agents, cache hit rate = cost.

5. Cluster: From One Machine to Many

When model or traffic exceeds single node: PD Separation — dedicated Prefill nodes (compute-optimized) and Decode nodes (bandwidth-optimized), transferring KV Cache via high-speed interconnect. Alibaba Cloud PAI-EAS MoE PD template: Prefill 8-GPU TP; Decode 8-way EP + 8-way DP; platform recommends ratios/parallelism per model/hardware/I/O lengths. Multimodal encoding can be separate. Large-scale parallelism & load balancing : Decode across dozens/hundreds of GPUs; expert load balance and inter-GPU comms critical. Elasticity & fault tolerance : autoscale on queue depth; reserve headroom; rapid request migration and recovery on GPU failure (common at scale). Single-node races on compute speed; cluster races on scheduling quality.

6. Sampling & Output: Controllable, Reproducible Results

Model outputs probability distribution; sampling picks next token. Parameters: Temperature (higher = more random), Top-p/Top-k (restrict candidate set). Same prompt can yield different answers due to sampling randomness. Even at temp=0, floating-point non-determinism from batch-order-dependent reduction can flip tokens. Some frameworks offer deterministic/batched-invariant modes (perf cost) for eval reproducibility and RL consistency. Two high-value features: Structured Output — compile schema (e.g., JSON) into grammar, mask invalid tokens each step, guaranteeing parseable tool calls. Speculative Decoding — small draft model proposes multiple tokens; large model verifies in parallel; mathematically preserves original distribution while speeding generation (speedup ∝ draft accuracy). Final tokens detokenized, parsed, streamed; usage logged for billing/monitoring.

9. Who Uses Inference Frameworks?

1. Model Vendors & Cloud Providers

Core infrastructure. Same GPUs, 2x efficiency = 2x users or 50% price. API price war = efficiency war. Dedicated optimization teams; deep customization or full custom stacks tuned to own models.

2. Enterprises Requiring Private Deployment

Finance, gov, healthcare, energy, manufacturing — data sovereignty/compliance. Direct framework users: select, deploy, tune, operate. Increasingly on domestic chips. Many buy integrated inference appliances/platforms to offload complexity, but understanding basics aids hardware sizing and capacity planning.

3. Agent & Application Teams

Mostly call APIs, don't deploy frameworks. But framework properties directly affect them: provider selection (TTFT, TBT, cache pricing), prompt/context design (cache-friendly = 10x cost diff), structured output support (tool reliability). Observability tools (e.g., Alibaba Cloud AgentLoop) trace per-step model/tool/retrieval calls, token usage, latency to pinpoint bottlenecks. Some teams self-host small models for intent classification/extraction to cut cost/latency — then they directly face framework choices.

4. Post-Training & Research Teams

Heavy users: RL needs massive trajectory generation; distillation needs teacher batch generation/scoring; eval runs thousands of tests; synthetic data needs bulk generation. Prioritize offline batch throughput and strict consistency between inference and training passes.

5. Individual Developers

Run models locally on laptops/desktops via llama.cpp/Ollama for learning, debugging, private data. Edge inference expanding to phones, PCs, cars. Needs: easy install, low resource footprint, runs on consumer hardware.

10. Conclusion

Inference frameworks sit between agents above and compute clusters below, serving every user query and agent call, and also powering RL sampling, evaluation, and data synthesis during training. As agents proliferate, call counts rise and contexts lengthen; every efficiency gain is multiplied by volume. Understanding inference frameworks is no longer just for algorithm engineers — it's essential for anyone building, deploying, or buying LLM-powered systems.