How vLLM Generates Tokens: A Deep Dive into the Source Code

Core Concepts and Process Overview

Three fixed assumptions are established before examining the code: (1) model weights are static read‑only data during inference, (2) the operating‑system process holds the weights and repeatedly executes the model, and (3) the GPU (or CPU) is the compute device, not a separate model daemon.

The vllm serve command creates two logical roles:

Frontend B – APIServer : handles HTTP, request templating, tokenization, and SSE framing.

Backend A – EngineCore + Worker : keeps weights resident, schedules requests, runs forward passes, and produces token_id outputs.

Both sides communicate via ZMQ: B → A sends EngineCoreRequest, A → B returns EngineCoreOutputs containing newly generated token IDs.

V1 Process Architecture

On a single‑node multi‑GPU setup the typical process count is:

API Server (default 1, adjustable with --api-server-count).

One EngineCore per data‑parallel (DP) rank.

One GPU Worker per device (count ≈ DP × TP × PP).

DP Coordinator appears only when DP > 1.

The main bottleneck is usually the KV cache stored in GPU memory; the API process becomes a secondary bottleneck only under very high QPS, multimodal loads, or many streaming connections.

Stage 1 – HTTP Entry

The request hits vllm/entrypoints/openai/api_server.py at the route POST /v1/chat/completions. The handler extracts the registered OpenAIServingChat instance from app.state and invokes its processing method.

The core business‑logic class lives in vllm/entrypoints/openai/chat_completion/serving.py. If streaming is requested, the handler wraps the async generator in a StreamingResponse; otherwise it returns a full ChatCompletionResponse JSON.

Stage 2 – Request Rendering and Engine Call

Inside OpenAIServingChat the following steps occur:

Apply the chat template, converting messages into prompt token_ids .

Build sampling parameters (temperature, max_tokens, stop strings, etc.).

Call self.engine_client.generate(...), which forwards to AsyncLLM.

If stream=true, a stream generator is returned.

At this point the system has performed "natural language → input token sequence & sampling config" but has not yet generated any output tokens.

Stage 3 – AsyncLLM Queuing and Cross‑Process Dispatch

The file vllm/v1/engine/async_llm.py defines generate(). Its responsibilities are:

Create a RequestOutputCollector for the request.

Construct an EngineCoreRequest from the input.

Register the request with the local OutputProcessor (which holds an incremental detokenizer).

Send the request to an independent EngineCore process via engine_core.add_request_async.

Yield RequestOutput objects from the queue back to the caller.

The key snippet:

output_processor.add_request(...)          # prepare detokenizer & queue
await engine_core.add_request_async(...)   # hand off to EngineCore

Stage 4 – ZMQ Transport

The client side lives in vllm/v1/engine/core_client.py. It holds two sockets: input_socket – pushes EngineCoreRequest to the engine. output_socket – pulls EngineCoreOutputs back.

On the same machine the default transport is ipc://; across machines or when IPC is unavailable tcp:// is used. This choice does not alter the division of labor between B (API) and A (Engine).

Stage 5 – EngineCore Computation

The core loop in vllm/v1/engine/core.py executes step(): scheduler.schedule(...) selects the batch of requests for this step (continuous batching). model_executor.execute_model(...) runs a forward pass on the shared weights and per‑request KV caches, producing logits.

Sampling (if needed) converts logits into new token_id s. scheduler.update_from_output(...) updates KV state and assembles EngineCoreOutputs.

Two phases are distinguished:

Prefill : the whole prompt (or chunked prompt) is fed, establishing a KV cache for the request.

Decode : typically one token is generated per step, appending to the KV cache.

The engine always outputs integer token_id s, not the final SSE string.

Stage 6 – Detokenization (Back to API)

The file vllm/v1/engine/output_processor.py runs an incremental detokenizer that turns token_ids into text deltas and wraps them in RequestOutput, which is placed into the request’s queue for the API side to consume.

Detokenization is deliberately performed on the API side to keep the worker focused on numeric computation and to simplify streaming protocol assembly.

Stage 7 – SSE Streaming

The streaming generator in vllm/entrypoints/openai/chat_completion/serving.py processes each RequestOutput as follows:

Compute the delta text relative to the previous frame.

Insert the delta into an OpenAI‑style ChatCompletionStreamResponse.

Yield data: {json}\n\n to the client.

When finished, yield data: [DONE]\n\n.

The client concatenates the received delta.content fields to render the final output.

Data‑Flow Transformation Table (Narrative)

Incoming JSON messages → tokenized to prompt token_ids (API tokenizer) → wrapped as EngineCoreRequest (sent via ZMQ) → processed by the engine to produce token_id logits → returned as EngineCoreOutputs → detokenized to text deltas (API) → streamed as SSE delta.content.

Concurrency Limiting Dimensions

The capacity hierarchy is:

KV / GPU memory : active sequences × average length set the hard concurrency ceiling.

Compute power : determines total tokens‑per‑second and per‑request inter‑token latency.

Scheduling parameters : max_num_seqs, max_num_batched_tokens, max_model_len.

API CPU : tokenization, multimodal loading, SSE packaging; mitigated with --api-server-count.

IPC / network : ZMQ is usually secondary on the same host; becomes noticeable across machines or over the public internet.

Typical bottleneck ordering:

KV/VRAM > GPU compute > scheduling > API CPU > ZMQ > WAN

. Data parallelism (DP) adds weight replicas and KV pools, increasing service concurrency; tensor parallelism (TP) mainly splits a large model across cards and does not linearly increase concurrent users.

Local Deployment Recommendations

vLLM does not officially support native Windows. Recommended environments:

WSL2 + Ubuntu + NVIDIA GPU (when a discrete GPU is present).

Native Linux.

CPU‑only PyTorch can be used for tiny test models but is not a baseline for multi‑user capacity.

Capacity Estimation Before Deployment

Rough formulas for weight and KV memory are provided, but real‑world budgeting must use the actual KV cache token count printed in the startup logs. Additional memory is consumed by CUDA context, activation peaks, and CUDA‑Graph overhead.

Typical Startup Parameter Templates

vllm serve <model> \
  --host 0.0.0.0 --port 8000 \
  --max-model-len 2048 \
  --max-num-seqs 2 \
  --gpu-memory-utilization 0.90

For a small internal team with a 24 GB GPU and a 7‑8 B model:

vllm serve <model> \
  --max-model-len 4096 \
  --max-num-seqs 8 \
  --gpu-memory-utilization 0.90

Multi‑card single‑model example:

vllm serve <model> --tensor-parallel-size 2

Verification and Benchmarking

Functional test with curl:

curl http://127.0.0.1:8082/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d "{\"model\":\"<model>\",\"messages\":[{\"role\":\"user\",\"content\":\"你好\"}],\"stream\":true}"

Capacity calibration uses vllm bench serve (or the historic benchmark_serving.py) to sweep concurrency and request rate, recording success rate, TTFT, ITL/TPOT, and output tokens‑per‑second. The maximum concurrency with near‑100 % success and acceptable latency is taken as the measured capacity; a safety factor of 0.7 is commonly applied for production.

Scaling Guidance

To relieve API‑CPU pressure, increase --api-server-count.

To raise service concurrency (more KV/weights), increase --data-parallel-size.

When a model does not fit a single card, enable TP/PP via --tensor-parallel-size (or pipeline parallel).

DP and TP can be combined: multiple DP replicas each with TP slicing.

Cross‑machine deployments require reachable tcp:// addresses and matching engine lifetimes.

Only the API (B) and engine (A) together constitute a runnable service; deploying the API alone is invalid.

Source‑Code Reading Order

vllm/entrypoints/openai/api_server.py

– routing and app state. vllm/entrypoints/openai/chat_completion/serving.py – OpenAIServingChat business logic. vllm/v1/engine/async_llm.py – generate and background output loop. vllm/v1/engine/core_client.py – ZMQ client implementation. vllm/v1/engine/core.py – step and request‑cancellation handling. vllm/v1/engine/output_processor.py – detokenization.

Documentation under docs/ for architecture details (paths vary by version).

All of the above illustrates the end‑to‑end flow from an HTTP chat request to streamed token output, exposing the critical points where memory, compute, and scheduling decisions affect latency and throughput.