How to Diagnose and Fix Common LLM Service Errors: Timeout, Rate Limiting, and OOM

Overview

After deployment, the three most frequent failure modes of large‑model services are timeout, rate limiting, and out‑of‑memory (OOM), accounting for more than 80% of incidents. In a production vLLM cluster running Qwen/Qwen3.5‑35B‑A3B‑FP8, over 200 alerts in two months showed 45% timeout, 30% OOM, and 15% rate‑limit events.

Systematic Troubleshooting Methodology

Observe the symptom (error code, log line, metric spike).

Locate the layer where the problem originates – client, API gateway, or inference engine.

Run the diagnostic commands listed for each layer.

Apply the specific remediation steps for the identified root cause.

Error Taxonomy

Timeout

HTTP 504 / client‑side timeout.

Causes: TTFT too high, low TPS, or network/connectivity failure.

Remediation: adjust three‑layer timeouts, limit prompt length, increase resources, enable chunked pre‑fill.

Rate Limiting

HTTP 429 Too Many Requests.

Causes: API‑gateway queue saturation, vLLM internal request queue.

Remediation: increase rate / burst in Nginx, reduce max‑num‑seqs, add token‑bucket limits on the client.

GPU OOM

Errors: torch.cuda.OutOfMemoryError or CUDA error: out of memory.

Causes: model too large for GPU memory, excessive concurrent sequences, very long prompts.

Remediation: lower gpu‑memory‑utilization, reduce max‑num‑seqs or max‑model‑len, use FP8 quantization, add more GPUs.

CPU OOM

System OOM killer kills the process.

Causes: model loading memory spike (~35 GB for a 35 B FP8 model), tokenizer memory leak, missing swap.

Remediation: add host RAM or swap, limit concurrent requests, restart service periodically.

Container OOMKilled

cgroup memory limit smaller than actual demand.

Remediation: raise resources.limits.memory in Kubernetes or --memory flag in Docker.

CUDA device‑side assert

Typical message: CUDA error: device‑side assert triggered.

Causes: corrupted model weights or NaN/Inf inputs.

Remediation: redownload model, verify with safetensors, run with CUDA_LAUNCH_BLOCKING=1 for a precise stack trace.

NCCL abort

Message: RuntimeError: NCCL communicator was aborted.

Causes: GPU topology issues, shared‑memory shortage, mismatched CUDA/NCCL versions.

Remediation: check nvidia‑smi topo -m, increase --shm-size, align NUMA affinity.

Timeout Diagnosis

1. Identify the Timeout Type

TTFT (Time‑to‑First‑Token) – measured by the TTFB field in curl timing output. High TTFT (>5 s) usually means long prompt or KV‑cache fragmentation.

TPS (Tokens‑per‑Second) – measured by the time between Total and TTFB. Low TPS indicates generation bottleneck.

Connection timeout – Connect > 1 s means network/connectivity issue.

curl -o /dev/null -s -w "
 DNS:%{time_namelookup}s
 Connect:%{time_connect}s
 TLS:%{time_appconnect}s
 TTFB:%{time_starttransfer}s
 Total:%{time_total}s
" -X POST http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"Qwen/Qwen3.5-35B-A3B-FP8","messages":[{"role":"user","content":"hello"}],"max_tokens":50}'

Decision rules:

If Connect > 1 s → network issue.

If TTFB > 5 s → TTFT problem – check prompt length and KV cache.

If Total − TTFB large → TPS slowdown – examine batch size and GPU load.

2. TTFT Root‑Cause Analysis

Baseline TTFT on a single A100 (FP8, gpu‑memory‑utilization=0.9) for Qwen/Qwen3.5‑35B‑A3B‑FP8:

Prompt Length   Normal TTFT   Warning Threshold   Critical Threshold
100 tokens      0.1‑0.3 s     0.5 s               1 s
1 000 tokens    0.3‑0.8 s     1.5 s               3 s
4 000 tokens    1‑3 s         5 s                 10 s
16 000 tokens   3‑8 s         15 s                30 s

KV‑Cache fragmentation – monitor vllm:gpu_cache_usage_perc. If it approaches 1.0, reduce gpu‑memory‑utilization or restart the service.

Long prompt pre‑fill – pre‑fill cost grows quadratically with token count. Use --enable-chunked-prefill (default in vLLM ≥ 0.7) or split/shorten prompts.

GPU saturation – check nvidia‑smi SM utilization > 95 %. Add more GPUs or lower concurrent sequences.

3. TPS Diagnosis

Baseline TPS on a single A100:

Concurrency   Per‑request TPS   Total TPS   Interpretation
1             60‑80             60‑80       Single‑request performance
10            30‑50             300‑500     Batching gains
50            15‑25             750‑1250    Near saturation
100           8‑15              800‑1500    Over‑provisioned, latency rises

If vllm:gpu_cache_usage_perc ≈ 1.0 → reduce max‑num‑seqs or increase gpu‑memory‑utilization.

If GPU temperature > 85 °C → improve cooling or lower workload.

If batch size too large → enable --enable-chunked-prefill or lower max‑num‑batched‑tokens.

Rate‑Limiting Diagnosis

Check response headers to see where the limit originates:

# Example response headers
Retry-After: 5
X-RateLimit-Limit: 100
X-RateLimit-Remaining: 0
X-RateLimit-Reset: 1710300000

If headers contain X‑RateLimit‑* → limit is enforced by the API gateway (Nginx/Envoy).

If response body contains the word vllm → limit is inside vLLM.

Nginx Rate‑Limit Configuration

# /etc/nginx/conf.d/llm‑rate.conf
limit_req_zone $binary_remote_addr zone=per_ip:10m rate=50r/s;
limit_req_zone $api_key zone=per_key:10m rate=20r/s;

server {
    listen 80;
    location /v1/chat/completions {
        limit_req zone=per_ip burst=20 nodelay;
        limit_req zone=per_key burst=10 nodelay;
        limit_req_status 429;
        proxy_pass http://vllm_backend;
    }
}

Token‑bucket algorithm explanation (included for completeness):

1. Bucket fills at a fixed <em>rate</em> (e.g., 50 tokens/s).
2. It can hold up to <em>burst</em> tokens for short spikes.
3. Each request consumes one token; if empty, the request is rejected with 429.

OOM Diagnosis

GPU OOM

Check GPU memory usage: nvidia‑smi -l 1 or

nvidia‑smi --query-gpu=memory.total,memory.used,memory.free --format=csv

.

Inspect vLLM logs for messages like torch.cuda.OutOfMemoryError or No available memory for the request.

Typical root causes:

Model too large for the GPU.

Excessive max‑num‑seqs causing KV‑cache overflow.

Very long prompts (e.g., 30 K tokens → ~7.5 GiB KV cache per request for Qwen3.5‑35B‑A3B‑FP8).

Remediation examples:

# Reduce KV‑cache pressure
python -m vllm.entrypoints.openai.api_server \
    --model Qwen/Qwen3.5-35B-A3B-FP8 \
    --gpu-memory-utilization 0.85 \
    --max-num-seqs 200 \
    --max-model-len 16384

CPU OOM

Check kernel OOM logs: dmesg | grep -i "oom\|killed".

Inspect process memory usage: ps aux --sort=-%mem | head -10.

Common causes: model loading spike (~35 GB for a 35 B FP8 model), tokenizer memory leak, missing swap.

Remediation: add swap (e.g.,

fallocate -l 32G /swapfile && mkswap /swapfile && swapon /swapfile

) or increase host RAM.

Container OOMKilled

Verify cgroup limit:

docker inspect <em>container</em> --format='{{.HostConfig.Memory}}'

.

Typical fix: raise resources.limits.memory in Kubernetes pod spec or --memory flag in Docker.

Case Study 1 – GPU OOM in Production

Timeline

03:05 – Alert: Process exited with code 137 (OOM Killed).

03:08 – Confirmed OOMKilled via docker inspect … --format '{{.State.OOMKilled}}'.

03:12 – dmesg shows OOM kill with 73 GiB RSS, exceeding the container limit of 72 GiB.

03:15 – vLLM logs reveal a burst of requests with prompt_tokens≈30 000 (≈7.5 GiB KV cache each).

Root Cause : max-model-len=32768 and max-num-seqs=256 allowed eight concurrent 30 K‑token prompts, exceeding the 54 GiB KV‑cache budget (GPU memory ≈ 80 GiB, 0.9 × usable for cache).

Mitigation

# Temporary fix – limit prompt length
docker run -d \
  --gpus "device=0" \
  --memory=72g \
  --shm-size=8g \
  --name vllm-server \
  vllm/vllm-openai:v0.7.6 \
  --model Qwen/Qwen3.5-35B-A3B-FP8 \
  --gpu-memory-utilization 0.9 \
  --max-num-seqs 200 \
  --max-model-len 16384   # lowered from 32768

Long‑run actions:

Stagger batch jobs to avoid simultaneous long prompts.

Add API‑gateway rule to reject prompts longer than 16 K tokens.

Increase container memory limit to 96 GiB.

Enable KV‑cache water‑mark alerts.

Case Study 2 – Intermittent Timeouts (14:00 – 14:30)

Symptoms : Timeout rate spikes from 0.1 % to 15 % every weekday afternoon.

Investigation :

Prometheus query shows QPS jumping from ~30 to ~150 during the window.

GPU utilization rises from 60 % to 100 % (GPU fully saturated).

nginx access logs reveal two batch‑worker IPs ( 10.0.2.55 and 10.0.2.56) generating >90 % of the traffic.

Both workers run a cron job daily_summary.sh at 14:00, producing thousands of summarisation requests.

Resolution :

Stagger the cron jobs by 30 minutes (one at 02:00, the other at 02:30).

Add client‑side concurrency limit (max 5 parallel requests) and rate limit (10 r/s per API key) in the summarisation script.

Introduce a dedicated Nginx rate‑limit zone for the batch‑worker subnet:

limit_req_zone $binary_remote_addr zone=per_ip:10m rate=30r/s;
limit_req_zone $api_key zone=per_key:10m rate=10r/s;

Post‑fix monitoring shows timeout rate back to <0.05 % for the following month.

Best‑Practice Checklist (Pre‑Deployment)

# pre_deploy_check.sh – run before starting vLLM
#!/bin/bash
echo "=== Pre‑deployment Checklist ==="
# 1. GPU driver version
DRIVER=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader | head -1)
if [[ $DRIVER =~ ^560|[6-9][0-9][0-9] ]]; then echo "GPU driver OK ($DRIVER)"; else echo "GPU driver too old ($DRIVER)"; fi
# 2. CUDA version
CUDA=$(nvcc --version 2>/dev/null | grep "release" | awk '{print $5}' | tr -d ',')
echo "CUDA version: $CUDA"
# 3. GPU memory size
nvidia-smi --query-gpu=name,memory.total --format=csv,noheader
# 4. System RAM
free -h | awk '/Mem:/ {print $2 " total, " $3 " used, " $4 " free"}'
# 5. Disk space for model directory
df -h /models/ | awk 'NR==2{print $4 " free in /models"}'
# 6. Model files integrity (safetensors)
MODEL_DIR="/models/Qwen3.5-35B-A3B-FP8"
if [[ -f $MODEL_DIR/config.json && -f $MODEL_DIR/tokenizer.json ]]; then echo "Model files OK"; else echo "Missing config or tokenizer"; fi
# 7. Docker version
docker --version
# 8. Port 8000 availability
ss -tlnp | grep -q ':8000' && echo "Port 8000 IN USE" || echo "Port 8000 AVAILABLE"
# 9. /dev/shm size
df -h /dev/shm | awk 'NR==2{print $2 " /dev/shm"}'
# 10. NUMA topology
nvidia-smi topo -m | head -5
echo "=== Check Complete ==="

Watchdog Script for Automatic OOM Recovery

#!/bin/bash
# vllm_watchdog.sh – monitors the service, restarts on OOM, sends alerts
VLLM_CMD="python -m vllm.entrypoints.openai.api_server \
    --model Qwen/Qwen3.5-35B-A3B-FP8 \
    --host 0.0.0.0 --port 8000 \
    --gpu-memory-utilization 0.88 \
    --max-num-seqs 200 \
    --max-model-len 16384"
HEALTH_URL="http://localhost:8000/health"
MAX_RESTARTS=5
RESTART_COUNT=0
COOLDOWN=60
log(){ echo "[$(date '+%Y-%m-%d %H:%M:%S')] $1"; }
check_gpu(){ nvidia-smi >/dev/null 2>&1 || { log "GPU check failed"; return 1; }; 
    temp=$(nvidia-smi --query-gpu=temperature.gpu --format=csv,noheader,nounits -i 0);
    [[ $temp -gt 90 ]] && { log "GPU temperature $temp°C too high"; return 1; }; return 0; }
start_vllm(){ log "Starting vLLM..."; $VLLM_CMD &> /var/log/vllm/vllm.log &
    VLLM_PID=$!
    for i in {1..12}; do
        curl -s $HEALTH_URL >/dev/null && { log "vLLM ready (PID $VLLM_PID)"; return 0; }
        sleep 10
done
    log "vLLM failed to become ready"; kill $VLLM_PID; return 1; }
while true; do
    check_gpu || { sleep $COOLDOWN; continue; }
    if start_vllm; then
        RESTART_COUNT=0
        while true; do
            sleep 15
            if ! kill -0 $VLLM_PID 2>/dev/null; then
                log "vLLM process $VLLM_PID died"; break; fi
            if ! curl -s --max-time 10 $HEALTH_URL >/dev/null; then
                log "Health check failed, killing vLLM"; kill -9 $VLLM_PID; break; fi
        done
    fi
    ((RESTART_COUNT++))
    if (( RESTART_COUNT >= MAX_RESTARTS )); then
        log "Maximum restarts reached, exiting watchdog"; exit 1; fi
    log "Restarting in $COOLDOWN seconds (attempt $RESTART_COUNT/$MAX_RESTARTS)"
    sleep $COOLDOWN
done

Systemd unit (save as /etc/systemd/system/vllm-watchdog.service)

# /etc/systemd/system/vllm-watchdog.service
[Unit]
Description=vLLM Watchdog Service
After=network.target nvidia-persistenced.service

[Service]
Type=simple
User=root
ExecStart=/opt/scripts/vllm_watchdog.sh
Restart=on-failure
RestartSec=30
StandardOutput=journal
StandardError=journal

[Install]
WantedBy=multi-user.target

Prometheus Metrics (vLLM ≥ 0.7)

# Request latency
histogram_quantile(0.99, sum(rate(vllm:e2e_request_latency_seconds_bucket[5m])) by (le))
# TTFT
histogram_quantile(0.99, sum(rate(vllm:time_to_first_token_seconds_bucket[5m])) by (le))
# TPS (time per output token)
histogram_quantile(0.50, sum(rate(vllm:time_per_output_token_seconds_bucket[5m])) by (le))
# GPU KV‑cache usage (0‑1)
vllm:gpu_cache_usage_perc
# CPU cache swap usage
vllm:cpu_cache_usage_perc
# Queue length
vllm:num_requests_waiting
# Running requests
vllm:num_requests_running
# Success / failure counters
rate(vllm:request_success_total[5m])
rate(vllm:request_failure_total[5m])
# Token throughput
rate(vllm:prompt_tokens_total[5m])
rate(vllm:generation_tokens_total[5m])

Prometheus Alert Rules (example)

# vllm_alerts.yml
groups:
- name: vllm_critical
  rules:
  - alert: VLLMGPUCacheNearFull
    expr: vllm:gpu_cache_usage_perc > 0.95
    for: 2m
    labels:
      severity: critical
    annotations:
      summary: "GPU KV cache usage > 95%"
      description: "Instance {{ $labels.instance }} is at {{ $value | humanizePercentage }} – risk of OOM"
  - alert: VLLMServiceDown
    expr: up{job="vllm"} == 0
    for: 1m
    labels:
      severity: critical
    annotations:
      summary: "vLLM service down"
      description: "Instance {{ $labels.instance }} has been unreachable for 1 minute"
  - alert: VLLMHighErrorRate
    expr: (sum(rate(vllm:request_failure_total[5m])) / (sum(rate(vllm:request_success_total[5m])) + sum(rate(vllm:request_failure_total[5m]))) > 0.05
    for: 5m
    labels:
      severity: critical
    annotations:
      summary: "vLLM request error rate > 5%"
      description: "Current error rate {{ $value | humanizePercentage }}"
- name: vllm_warning
  rules:
  - alert: VLLMHighTTFT
    expr: histogram_quantile(0.99, sum(rate(vllm:time_to_first_token_seconds_bucket[5m])) by (le)) > 5
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: "TTFT P99 > 5s"
  - alert: VLLMQueueBacklog
    expr: vllm:num_requests_waiting > 50
    for: 3m
    labels:
      severity: warning
    annotations:
      summary: "Request queue backlog > 50"
  - alert: GPUTemperatureHigh
    expr: nvidia_smi_temperature_gpu > 83
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: "GPU temperature > 83°C"
  - alert: VLLMKVCacheSwap
    expr: vllm:cpu_cache_usage_perc > 0
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: "KV cache swapped to CPU – TPS may drop"

Monitoring Dashboard Recommendations (Grafana)

Row 1 – Overview : service up/down, current QPS, success rate, waiting queue length.

Row 2 – Latency : TTFT P50/P95/P99, end‑to‑end latency P50/P95/P99, per‑token generation speed.

Row 3 – Resources : GPU utilization, GPU temperature, GPU memory used, KV‑cache usage, CPU memory, swap usage.

Row 4 – Throughput : prompt tokens/s, generation tokens/s, running vs waiting requests.

Key vLLM Launch Configurations

Medium‑load profile (single A100 80 GB):

python -m vllm.entrypoints.openai.api_server \
    --model Qwen/Qwen3.5-35B-A3B-FP8 \
    --host 0.0.0.0 --port 8000 \
    --gpu-memory-utilization 0.88 \
    --max-num-seqs 200 \
    --max-model-len 16384 \
    --enable-chunked-prefill

High‑throughput profile (more concurrency, lower latency):

python -m vllm.entrypoints.openai.api_server \
    --model Qwen/Qwen3.5-35B-A3B-FP8 \
    --gpu-memory-utilization 0.92 \
    --max-num-seqs 512 \
    --max-num-batched-tokens 65536 \
    --max-model-len 8192 \
    --enable-chunked-prefill

Low‑latency profile (few concurrent requests, larger context):

python -m vllm.entrypoints.openai.api_server \
    --model Qwen/Qwen3.5-35B-A3B-FP8 \
    --gpu-memory-utilization 0.85 \
    --max-num-seqs 32 \
    --max-num-batched-tokens 8192 \
    --max-model-len 32768

Client‑Side Timeout Settings (OpenAI SDK example)

import openai
client = openai.OpenAI(
    base_url="http://llm.example.com/v1",
    api_key="not‑needed",
    timeout=openai.Timeout(
        connect=10.0,   # connection timeout
        read=180.0,    # overall response timeout
        write=30.0,    # request send timeout
        pool=10.0      # connection‑pool timeout
    ),
    max_retries=3,
)
response = client.chat.completions.create(
    model="Qwen/Qwen3.5-35B-A3B-FP8",
    messages=[{"role": "user", "content": "写一首诗"}],
    max_tokens=500,
    timeout=180,
)

Graceful Degradation (Fallback Model Router)

from dataclasses import dataclass
import requests, time, logging
logger = logging.getLogger(__name__)

@dataclass
class ModelEndpoint:
    name: str
    url: str
    model: str
    priority: int   # lower = higher priority
    max_tokens: int
    healthy: bool = True
    last_check: float = 0.0
    failure_count: int = 0

class ModelRouter:
    def __init__(self):
        self.endpoints = [
            ModelEndpoint("primary", "http://gpu‑01:8000", "Qwen/Qwen3.5-35B-A3B-FP8", 1, 4096),
            ModelEndpoint("secondary", "http://gpu‑02:8000", "Qwen/Qwen3.5-35B-A3B-FP8", 2, 4096),
            ModelEndpoint("fallback", "http://gpu‑03:8000", "Qwen/Qwen‑Coder-30B-A3B-FP8", 3, 2048),
        ]
    def get_healthy(self):
        healthy = [e for e in self.endpoints if e.healthy]
        if not healthy:
            logger.critical("All endpoints unhealthy – forcing primary")
            self.endpoints[0].healthy = True
            return self.endpoints[0]
        return sorted(healthy, key=lambda e: e.priority)[0]
    def mark_unhealthy(self, ep):
        ep.healthy = False
        ep.failure_count += 1
        logger.warning(f"{ep.name} marked unhealthy (failures={ep.failure_count})")
    def mark_healthy(self, ep):
        ep.healthy = True
        ep.failure_count = 0
    def health_check_all(self):
        for ep in self.endpoints:
            try:
                r = requests.get(f"{ep.url}/health", timeout=5)
                if r.status_code == 200 and not ep.healthy:
                    logger.info(f"{ep.name} recovered")
                    self.mark_healthy(ep)
                elif r.status_code != 200:
                    self.mark_unhealthy(ep)
            except Exception:
                self.mark_unhealthy(ep)

Summary

By following the three‑layer troubleshooting workflow, using the concrete diagnostic commands and configuration snippets above, operators can quickly locate the root cause of timeouts, rate limits, or OOM events, apply targeted remediation (adjusting max‑num‑seqs, gpu‑memory‑utilization, prompt length limits, or Nginx rate‑limit settings), and keep large‑language‑model services reliable and performant.