How to Deploy vLLM for an OpenAI‑Compatible Inference Service

1. Define Service Boundaries

vLLM handles model loading, batching, KV cache, and inference endpoints, but should not directly manage public TLS, enterprise authentication, tenant quotas, or cross‑instance load balancing. Typically vLLM listens on a loopback or internal address while Nginx/API Gateway provides a unified entry point. Before launch, confirm model precision, maximum context length, GPU count, tensor‑parallel degree, maximum concurrency, I/O limits, model logical name, chat template, and access scope; without these boundaries, memory and concurrency parameters cannot be configured sensibly.

2. Verify Host, GPU, and Container Runtime

Run nvidia-smi --query-gpu=… --format=csv, docker version, and docker compose version to ensure the host driver and Docker are compatible. The host’s nvidia-smi success does not guarantee container GPU access; use a trusted CUDA image to run nvidia-smi inside the container. If the container fails, first check the NVIDIA Container Toolkit, Docker device request, and daemon logs before altering vLLM parameters.

3. Validate Model Directory

Inspect <model_dir>/config.json and tokenizer_config.json with jq to extract

{architectures,model_type,torch_dtype,max_position_embeddings}

and {model_max_length,chat_template}. Verify that required files (e.g., *.safetensors, tokenizer.json) exist and that the directory structure points to the correct locations. When shared storage loads slowly, combine iostat, storage throughput, and container logs to diagnose; low GPU utilization does not necessarily mean the model process is stuck.

4. Confirm Image and Parameters

Inspect the vLLM image digest with docker image inspect and check the version via docker run --entrypoint vllm <image> --version. Use vllm serve --help to list supported flags. Ensure that host, port, served-model-name, tensor-parallel-size, dtype, max-model-len, gpu-memory-utilization, and max-num-seqs match the current image version; mixing commands from different versions is prohibited.

5. Minimal Startup (Foreground)

Example for GPUs 0 and 1 with tensor parallelism 2 and BF16 dtype:

docker run --rm --gpus "device=0,1" \
  --ipc=host --shm-size=32g \
  -p 8000:8000 -v "<model_dir>:/models/model:ro" \
  --entrypoint vllm <vllm_image>:<tag> \
  serve /models/model \
  --served-model-name <external_model_name> --host 0.0.0.0 --port 8000 \
  --tensor-parallel-size 2 --dtype bfloat16 --max-model-len 8192 \
  --gpu-memory-utilization 0.85 --max-num-seqs 32

The gpu-memory-utilization flag controls how much GPU memory is allocated for the model and KV cache; setting it too high can cause OOM during CUDA graphs, communication buffers, or peak request bursts. Enable trust-remote-code only after code review and when the model explicitly requires it.

6. Solidify Deployment with Docker Compose

Compose file snippet:

services:
  vllm-api:
    image: <vllm_image>:<tag>
    container_name: vllm_api_8000
    restart: unless-stopped
    ipc: host
    shm_size: 32g
    ports:
      - "127.0.0.1:8000:8000"
    environment:
      CUDA_VISIBLE_DEVICES: "0,1"
    volumes:
      - <model_dir>:/models/model:ro
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              device_ids: ["0", "1"]
                capabilities: [gpu]
    entrypoint: ["vllm", "serve"]
    command:
      - /models/model
      - --served-model-name
      - <external_model_name>
      - --host
      - 0.0.0.0
      - --port
      - "8000"
      - --tensor-parallel-size
      - "2"
      - --dtype
      - bfloat16
      - --max-model-len
      - "8192"
      - --gpu-memory-utilization
      - "0.85"
      - --max-num-seqs
      - "32"
    healthcheck:
      test: ["CMD-SHELL", "python3 -c \"import urllib.request; urllib.request.urlopen('http://127.0.0.1:8000/health', timeout=3)\""]
      interval: 30s
      timeout: 5s
      retries: 5
      start_period: 600s
    logging:
      driver: json-file
      options:
        max-size: 100m
        max-file: "5"

Before starting, render the configuration, verify the port is free, and check GPU usage. Note that the Compose service name ( vllm-api) is used, not the container name.

7. Layered API Verification

Check the listening port and health endpoint:

ss -lntp | grep ':8000 '
curl -fsS --connect-timeout 2 --max-time 10 http://127.0.0.1:8000/health

Confirm the model ID matches served-model-name:

curl -fsS http://127.0.0.1:8000/v1/models | jq .

Test a non‑streaming chat completion to verify response structure, finish_reason, and usage:

curl -fsS http://127.0.0.1:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model":"<external_model_name>",
    "messages":[{"role":"user","content":"only reply ready"}],
    "temperature":0,
    "max_tokens":16,
    "stream":false
  }' | jq .

For streaming, use curl -N and ensure data arrives chunk‑by‑chunk rather than being cached by a proxy.

8. SDK Compatibility Check

import os
from openai import OpenAI

client = OpenAI(
    base_url=os.getenv("LLM_BASE_URL", "http://127.0.0.1:8000/v1"),
    api_key=os.getenv("LLM_API_KEY", "local-not-checked"),
)

response = client.chat.completions.create(
    model="<external_model_name>",
    messages=[{"role":"user","content":"return whether the current request succeeded."}],
    temperature=0,
    max_tokens=64,
)
print(response.choices[0].message.content)

The API key must be injected via environment variables or a secret manager; it should never be committed to the repository. Acceptance testing should also cover oversized inputs, malformed structures, streaming termination, client interruption, and concurrent request behavior.

9. Chat Template and Base Model

If the base model or tokenizer lacks a chat_template, Chat Completions may fail or produce malformed output. The template must come from the official model release or an internally validated version and be version‑controlled. Changing the template alters all prompts, so it constitutes a model‑behavior change that requires regression testing and the ability to roll back to the previous template.

10. Concurrency and Memory Baseline

Increasing max-model-len raises the KV cache per request; increasing max-num-seqs raises concurrency and queueing behavior. The following script performs a short‑request stability check (not a professional benchmark):

#!/usr/bin/env bash
set -euo pipefail

URL="http://127.0.0.1:8000/v1/chat/completions"
MODEL="<external_model_name>"
TOTAL="32"
CONCURRENCY="8"
TMP_DIR="$(mktemp -d)"
trap 'rm -rf "$TMP_DIR"' EXIT

request_one() {
  id="$1"
  curl -sS -o "$TMP_DIR/$id.json" -w "id=$id code=%{http_code} total=%{time_total}
" \
    -H 'Content-Type: application/json' \
    -d "{\"model\":\"$MODEL\",\"messages\":[{\"role\":\"user\",\"content\":\"return number $id\"}],\"temperature\":0,\"max_tokens\":32}" \
    "$URL"
}
export -f request_one
seq 1 "$TOTAL" | xargs -P "$CONCURRENCY" -I{} bash -c 'request_one "$@"' _ {}

During load testing, collect GPU and service metrics synchronously. vLLM exposes Prometheus metrics whose names and labels may vary across versions. When throughput stops increasing while queue latency grows, the instance has exceeded a reasonable concurrency boundary; the correct response is to rate‑limit or add instances, not to raise max-num-seqs further.

11. Startup Failures and OOM

When a container exits, first inspect the exit code, OOM flag, logs, and the actual command before repeatedly restarting. For CUDA OOM, query each GPU’s processes and container mappings. To troubleshoot, lower gpu-memory-utilization, max-model-len, or max-num-seqs one variable at a time and repeat the same load test.

12. Multi‑GPU and NCCL Troubleshooting

If multi‑GPU initialization hangs, verify tensor‑parallel size, visible GPUs, topology, processes, and Xid errors:

nvidia-smi topo -m
ps -eo pid,ppid,stat,etime,cmd | grep -E '[v]llm|[t]orch'
journalctl -k --since '-30 min' --no-pager | grep -Ei 'NVRM|Xid|nvlink|pcie|oom' || true

Enable NCCL debugging only for a short reproducing window to avoid excessive logs:

NCCL_DEBUG=INFO NCCL_DEBUG_SUBSYS=INIT,GRAPH CUDA_VISIBLE_DEVICES=0,1 vllm serve <model_dir> \
  --tensor-parallel-size 2 \
  --served-model-name <external_model_name> \
  --port 8000

13. Expose Service via Nginx

upstream vllm_backend {
  server 127.0.0.1:8000;
  keepalive 64;
}

location /v1/ {
  proxy_pass http://vllm_backend;
  proxy_http_version 1.1;
  proxy_set_header Connection "";
  proxy_set_header X-Request-ID $request_id;
  proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
  proxy_connect_timeout 3s;
  proxy_send_timeout 60s;
  proxy_read_timeout 600s;
  proxy_request_buffering off;
  proxy_buffering off;
  gzip off;
}

Backup the Nginx config, test syntax, reload, and verify the model list with curl -fsS https://<api_domain>/v1/models | jq .. The vLLM port should never be exposed publicly; authentication, TLS, rate limiting, request‑body limits, and audit logging must be handled by a controlled gateway, and logs must avoid recording full prompts or Authorization headers.

14. Upgrade and Rollback

Backup the current Compose file, render it, and record the image inspect JSON and model config checksum. Deploy the new version on a new port and run non‑streaming, streaming, long‑context, and gray‑release tests before cutting over. If the new version fails, first revert the gateway configuration, then stop the new instance after draining in‑flight requests. Successful rollback restores the endpoint, model name, chat template, core request behavior, streaming output, error rate, and latency.

15. Daily Health Checks

#!/usr/bin/env bash
set -euo pipefail

CONTAINER="vllm_api_8000"
BASE_URL="http://127.0.0.1:8000"
EXPECTED_MODEL="<external_model_name>"

if [[ "$(docker inspect -f '{{.State.Running}}' "$CONTAINER")" == "true" ]]; then
  curl -fsS --max-time 5 "$BASE_URL/health" > /dev/null
  MODELS=$(curl -fsS --max-time 10 "$BASE_URL/v1/models")
  echo "$MODELS" | jq -e --arg model "$EXPECTED_MODEL" '.data[] | select(.id == $model)' > /dev/null
  nvidia-smi --query-gpu=index,utilization.gpu,memory.used,memory.total,temperature.gpu --format=csv
fi

Before auto‑restarting, distinguish between slow model loading, OOM, NCCL/Xid errors, gateway failures, and overload. A reliable production rollout also covers image digest verification, read‑only model mounts, GPU/TP consistency, context and concurrency limits, SSE handling, authentication, log sanitization, monitoring, rate limiting, and rollback procedures.