Inference Frameworks vs Platforms: How LLMs Actually Run on Your Hardware

Introduction

You have a GPU and a downloaded model — what software do you run next? Tutorials mention Ollama, LM Studio, vLLM, SGLang, and they often call each other. For example, LM Studio may use llama.cpp underneath. Understanding why these names appear together starts with separating their responsibilities.

Previous articles covered GPU compute, memory, bandwidth, and the inference steps of loading, prefill, and decode. Model files hold trained weights; hardware provides compute and storage. Software in the middle reads the model, orchestrates computation, and returns generated output. Inference frameworks and inference platforms play distinct roles here.

Inference Frameworks vs Platforms: Separate Concerns

Inference frameworks (also called inference engines) run the model: load weights, manage runtime data, invoke hardware for prefill and decode. Inference platforms make models usable and manageable: handle downloading, starting, switching, and exposing APIs so callers don't deal with low-level details.

The article groups tools by primary responsibility:

Platforms & run tools — focus on convenient use, management, deployment. Examples: Ollama, LM Studio, Lemonade, Xinference.

Frameworks & engines — focus on efficient model computation. Examples: llama.cpp, vLLM, SGLang, TokenSpeed.

These categories overlap; a platform can bundle an engine, and an engine can expose its own API (llama.cpp and vLLM both serve directly).

Layered View of a Local Inference Request

User or business application
  ↓
Use & management layer: select model, set parameters, receive requests
  ↓
Inference execution layer: load weights, manage cache, organize computation
  ↓
Compute backend & drivers: dispatch compute to specific hardware
  ↓
CPU, GPU, etc.: execute compute, access data

Each layer doesn't require separate software; platforms can integrate engines, engines can provide APIs.

Inference Frameworks: Organizing Model Computation

Model Loading: Interpreting Files, Placing Weights

Model files aren't directly executable. Frameworks must recognize the architecture, read weights, and allocate memory. Whether the model runs entirely on GPU or splits across CPU/GPU depends on engine support and hardware. Some engines support hybrid inference: when VRAM is insufficient, part of the weights stay in system memory for CPU processing — at the cost of speed. After download, you must verify the engine supports the model's structure, file format, and quantization.

Prefill & Decode: Invoking the Right Kernels

User input becomes tokens (not necessarily one per character/word). Prefill processes all input tokens at once; decode generates subsequent tokens step by step. The framework schedules computation, allocates temporary buffers, and manages KV cache — reusable intermediate attention results, not final answers. Low-level matrix multiplication and attention kernels are selected based on hardware and precision; different kernel implementations yield different performance on the same GPU and model.

Handling Concurrent Requests: Reducing Queueing and Idle Resources

If ten users ask simultaneously, sequential processing queues later users and underutilizes the GPU. vLLM and similar frameworks use continuous batching : multiple requests compute together, with new requests inserted and finished ones removed between steps, without waiting for the whole batch to finish. This improves overall throughput (requests per second) but doesn't necessarily reduce per-request latency.

Inference Platforms: Turning Compute into Usable Services

Using an engine directly still requires manual model download, configuration, and service startup. Platforms centralize these operations, offering chat UI, CLI, or API access. They don't add model knowledge but save configuration effort. Four tools illustrate different emphases:

Ollama: Simplify Model Running & App Integration

Ollama bundles model download, execution, and API serving. You can chat via UI/CLI or let your program call its service. It reduces service setup for local-first workflows, but you must still confirm the version supports your model's architecture and format.

LM Studio: Visual Interface for Local Models

LM Studio provides a desktop GUI with model search, download, chat, parameter tuning, and API exposure. Its relationship with llama.cpp is direct: GGUF models selected in LM Studio are often executed by llama.cpp. On Apple Silicon Macs, LM Studio also supports Apple's MLX framework. These execution components are called runtimes ; the user operates the UI while the integrated runtime does the math — no need to choose between LM Studio and llama.cpp.

Lemonade: Unified Local Inference Backend Management

Lemonade manages models and inference backends in a single local service. It can run inference via llama.cpp, and on supported devices/models, via FastFlowLM, Ryzen AI, and other backends. Applications call one endpoint to access different backends, including NPU acceleration. NPU support depends on chip, backend, and model format alignment — having an NPU doesn't mean any model can use it.

Xinference: Centralized Deployment & Management of Multi-Model Services

Xinference targets model service deployment and management, supporting chat, embedding, reranking, speech, and more, scaling from single-node to multi-node. A knowledge-base app might use an embedding model for retrieval, a reranker for ordering, and a chat model for answers — Xinference manages them all under one API gateway. It integrates vLLM, SGLang, llama.cpp, etc.; Xinference handles service orchestration while the chosen engine executes inference.

Common Inference Engines: What Each Optimizes For

Running one model on a laptop vs. serving many requests on a server demands different software. Four engines share the compute responsibility but prioritize differently:

llama.cpp: Run Models on Diverse Hardware

Implemented in C/C++, llama.cpp is ubiquitous for GGUF models. It supports multiple hardware backends and CPU/GPU hybrid inference, usable standalone or embedded. On personal computers running quantized models and tuning CPU/GPU split, you'll often encounter llama.cpp. Despite the name, it supports many model families beyond Llama.

vLLM: Maximize Overall Serving Throughput

vLLM provides both inference execution and serving capabilities, common in server deployments. Its PagedAttention partitions KV cache into blocks, reducing VRAM fragmentation. With continuous batching, multiple requests share compute resources more effectively. Efficient cache space allocation matters as much as raw compute speed when many requests run concurrently.

SGLang: Low Latency & Cache Reuse

SGLang targets model serving with prefix caching and multi-GPU parallelism, focusing on response latency and throughput. Example: repeated requests sharing the same system prompt can reuse the prefill computation of that prefix (cached intermediate results), skipping redundant prefill. The suffix still generates fresh. vLLM also supports cache reuse; comparing engines requires looking at concrete implementations, model/hardware fit, not just feature lists.

TokenSpeed: Emerging Engine for Agent Workloads

TokenSpeed optimizes for agent scenarios — request scheduling, KV cache management, and kernel tuning. Agents repeatedly call the model, execute tools, then call again; each round's wait accumulates. TokenSpeed targets this pattern's inference efficiency. Benchmark results on specific hardware/model combos don't transfer directly; verify version support for your model and hardware before adopting.

Why Switching Platforms Doesn't Guarantee Speedups

If two platforms use similar versions of the same engine, with identical model, hardware, and settings, merely changing the UI won't yield noticeable acceleration. Conversely, even with the same model name, different quantization, context length, GPU offload ratio, or cache settings alter speed and memory usage. Switching engines also requires confirming support for the model's structure and format — a GGUF file may not run everywhere unchanged. Meaningful comparison means checking what actually runs, then measuring time-to-first-token, generation speed, and concurrency handling.

Conclusion

Choose local tools by usage pattern. For casual chat and model testing, prioritize ease of operation and device compatibility — compare Ollama, LM Studio, Lemonade. For centralized multi-model service management, evaluate Xinference. When the model runs but you face long first-token latency, slow generation, or queueing under load, investigate the underlying engine and its configuration. Platforms get the service running; how the engine drives the hardware is the next layer to analyze for performance issues.