How to Configure Nginx Load Balancing for Multiple LLM Instances

1. Determine What Can Be Balanced

Load balancing must operate only between "equivalent backends". The model name, API schema, context limits, and chat template must be identical across instances; otherwise HTTP 200 responses can hide divergent output. A checklist of required conditions and consequences of mismatches includes model logical name, API structure, context limits, chat template, GPU capacity, and timeout strategy.

Before adding Nginx, each backend is probed directly with a Bash script that checks /health and /v1/models endpoints, listing any failures.

#!/usr/bin/env bash
set -euo pipefail

BACKENDS=(
  "http://10.20.0.11:8000"
  "http://10.20.0.12:8000"
  "http://10.20.0.13:8000"
)

for backend in "${BACKENDS[@]}"; do
  echo "Checking ${backend}"
  curl -fsS --connect-timeout 2 --max-time 5 "${backend}/health" >/dev/null
  curl -fsS --connect-timeout 2 --max-time 10 "${backend}/v1/models" | jq -r '.data[].id'
 done

Even if model names match, a low‑randomness request should be sent to verify response structure because floating‑point differences, parallel strategies, or sampling can still cause output drift.

for host in 10.20.0.11 10.20.0.12 10.20.0.13; do
  curl -fsS "http://${host}:8000/v1/chat/completions" \
    -H 'Content-Type: application/json' \
    -d '{
      "model":"<模型名>",
      "messages":[{"role":"user","content":"只回复 ready"}],
      "temperature":0,
      "max_tokens":16,
      "stream":false
    }' | jq -c '{model,finish_reason:.choices[0].finish_reason,usage}'
 done

2. Check Nginx Version and Build Options

Different distributions ship different modules and default paths. Record the Nginx version, compile flags, and loaded configuration before deployment, especially TLS, real‑IP, and status modules.

nginx -v
nginx -V 2>&1 | tr ' ' '
' | sort
sudo nginx -T > /tmp/nginx-effective-config.txt

The output may contain internal domain names, certificate paths, or credentials; handle it as sensitive data.

3. Choose a Distribution Algorithm

Nginx open‑source supports round_robin, least_conn, ip_hash, and hash with consistent. Because LLM inference latency varies from seconds to minutes, simple round‑robin (which only counts request numbers) is insufficient. least_conn prefers nodes with fewer active connections, which works well for homogeneous instances.

For heterogeneous instances, weight can be assigned based on capacity tests rather than raw GPU memory.

upstream llm_backend {
    least_conn;
    zone llm_backend 64k;
    server 10.20.0.11:8000 max_fails=2 fail_timeout=20s;
    server 10.20.0.12:8000 max_fails=2 fail_timeout=20s;
    server 10.20.0.13:8000 max_fails=2 fail_timeout=20s;
    keepalive 64;
}

Weighted example:

upstream llm_backend_weighted {
    least_conn;
    zone llm_backend_weighted 64k;
    server 10.20.0.21:8000 weight=2 max_fails=2 fail_timeout=20s;
    server 10.20.0.22:8000 weight=1 max_fails=2 fail_timeout=20s;
    keepalive 64;
}

4. Proxy Configuration for Streaming and Non‑Streaming APIs

OpenAI‑compatible endpoints use normal HTTP POST; streaming responses use Server‑Sent Events (SSE). Critical settings: disable response buffering, set generous read timeouts, and forward request IDs.

server {
    listen 443 ssl http2;
    server_name <API域名>;
    ssl_certificate <证书路径>;
    ssl_certificate_key <私钥路径>;
    client_max_body_size 10m;
    client_body_timeout 30s;

    location /v1/ {
        proxy_pass http://llm_backend;
        proxy_http_version 1.1;
        proxy_set_header Connection "";
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header X-Request-ID $request_id;
        proxy_connect_timeout 3s;
        proxy_send_timeout 60s;
        proxy_read_timeout 600s;
        proxy_request_buffering off;
        proxy_buffering off;
        gzip off;
        proxy_next_upstream error timeout http_502 http_503 http_504;
        proxy_next_upstream_tries 2;
        proxy_next_upstream_timeout 5s;
    }
}

proxy_read_timeout 600s

limits idle time between upstream reads, not the total inference duration; it must be tuned to the longest expected generation time.

5. Separate Health Checks from Business Traffic

Health endpoints should not be placed under the generic /v1/ path, otherwise a healthy gateway could mask a dead backend. Add a lightweight local health endpoint:

location = /gateway-health {
    access_log off;
    default_type text/plain;
    return 200 "ok
";
}

Monitoring should combine gateway health, per‑instance health, and low‑frequency synthetic requests.

6. Logging that Forms an Evidence Chain

The default combined log lacks upstream address and timing. Define a JSON log format that records request ID, status, upstream address, and various timing fields while omitting request bodies and credentials.

log_format llm_json escape=json '{'
    "time":"$time_iso8601",
    "request_id":"$request_id",
    "remote_addr":"$remote_addr",
    "method":"$request_method",
    "uri":"$uri",
    "status":$status,
    "request_length":$request_length,
    "bytes_sent":$bytes_sent,
    "request_time":$request_time,
    "upstream_addr":"$upstream_addr",
    "upstream_status":"$upstream_status",
    "upstream_connect_time":"$upstream_connect_time",
    "upstream_header_time":"$upstream_header_time",
    "upstream_response_time":"$upstream_response_time"'
}';
access_log /var/log/nginx/llm_access.json llm_json;
error_log /var/log/nginx/llm_error.log warn;

Example jq queries aggregate request counts per node, identify long‑running requests, and correlate request IDs with backend logs.

7. Rate Limiting, Connection Limits, and Overload Protection

limit_req

counts requests but does not understand token cost; it is suitable for burst protection but must be combined with business‑level quotas.

map $http_x_tenant_id $tenant_key {
    default $http_x_tenant_id;
    "" $binary_remote_addr;
}
limit_req_zone $tenant_key zone=llm_per_tenant:20m rate=2r/s;
limit_conn_zone $tenant_key zone=llm_conn_per_tenant:20m;

server {
    location /v1/ {
        limit_req zone=llm_per_tenant burst=10 nodelay;
        limit_conn llm_conn_per_tenant 8;
        proxy_pass http://llm_backend;
    }
}
limit_req_status 429;
limit_conn_status 429;

Clients receiving 429 should implement exponential back‑off with jitter.

8. Session‑Based Consistent Hashing (When Appropriate)

If the backend caches prefixes and many multi‑turn conversations share a stable prefix, consistent hashing on a generated X-Session-ID can improve cache hit rates, but it does not synchronize KV caches and can create hotspots.

upstream llm_session_backend {
    zone llm_session_backend 64k;
    hash $http_x_session_id consistent;
    server 10.20.0.11:8000 max_fails=2 fail_timeout=20s;
    server 10.20.0.12:8000 max_fails=2 fail_timeout=20s;
    server 10.20.0.13:8000 max_fails=2 fail_timeout=20s;
    keepalive 64;
}

9. Configuration Changes Must Follow a Closed Loop

Never edit production config and restart directly. Use a backup, generate a new file, run nginx -t, reload, verify, and be ready to roll back.

#!/usr/bin/env bash
set -euo pipefail
NGINX_DIR="/etc/nginx"
BACKUP_ROOT="<配置备份目录>"
STAMP="$(date +%Y%m%d-%H%M%S)"
BACKUP_DIR="${BACKUP_ROOT}/nginx-${STAMP}"

sudo install -d -m 0700 "${BACKUP_DIR}"
sudo cp -a "${NGINX_DIR}/." "${BACKUP_DIR}/"
sudo nginx -t

echo "Backup created: ${BACKUP_DIR}"

After changes, diff the files, test, then reload with systemctl reload nginx. Verify TLS, model list, and both streaming and non‑streaming endpoints using --resolve to force DNS to the gateway IP.

10. Instance Maintenance and Connection Drain

Before taking a backend offline, mark it down or temporarily remove it from the upstream, run syntax check, and reload. Existing connections continue on the old worker until they finish.

upstream llm_backend {
    least_conn;
    zone llm_backend 64k;
    server 10.20.0.11:8000 max_fails=2 fail_timeout=20s;
    server 10.20.0.12:8000 down;
    server 10.20.0.13:8000 max_fails=2 fail_timeout=20s;
    keepalive 64;
}

Monitor logs to ensure no new requests reach the drained node, then stop the model instance during low traffic.

11. Common Fault Diagnosis

502 Bad Gateway : Verify TCP connectivity, port listening, and that the health endpoint responds. Only attribute to a dead backend if logs contain "connect() failed (111: Connection refused)".

ip route get 10.20.0.11
nc -vz -w 2 10.20.0.11 8000
curl -v --connect-timeout 2 --max-time 10 http://10.20.0.11:8000/health
sudo grep -F '10.20.0.11:8000' /var/log/nginx/llm_error.log | tail -n 50

504 Gateway Timeout : Correlate request ID across Nginx and backend logs to see if the delay is due to long queue, generation, or GPU issues. Blindly increasing proxy_read_timeout hides the root cause.

REQUEST_ID="<请求ID>"
sudo grep -F "${REQUEST_ID}" /var/log/nginx/llm_access.json
sudo grep -F "${REQUEST_ID}" /var/log/nginx/llm_error.log
ssh <后端主机> "journalctl -u <模型服务名> --since '-15 min' --no-pager | grep -F '${REQUEST_ID}'"

Streaming returns as a single response : Compare direct backend curl -N output with the proxied request to identify buffering or compression in the path.

12. Automated Inspection of Multiple Nodes

A Bash script checks Nginx syntax, gateway health, each backend's health endpoint, and model name consistency. It does not auto‑reload or remove nodes; alerts are raised for manual handling.

#!/usr/bin/env bash
set -euo pipefail
EXPECTED_MODEL="${EXPECTED_MODEL:-<模型名>}"
GATEWAY_URL="${GATEWAY_URL:-https://<API域名>}"
BACKENDS=(
  "http://10.20.0.11:8000"
  "http://10.20.0.12:8000"
  "http://10.20.0.13:8000"
)

sudo nginx -t >/dev/null
curl -fsS --max-time 5 "${GATEWAY_URL}/gateway-health" >/dev/null

failed=0
for backend in "${BACKENDS[@]}"; do
  if ! curl -fsS --max-time 5 "${backend}/health" >/dev/null; then
    echo "CRITICAL: health failed ${backend}" >&2
    failed=1
    continue
  fi
  if ! curl -fsS --max-time 10 "${backend}/v1/models" |
       jq -e --arg m "${EXPECTED_MODEL}" '.data[] | select(.id == $m)' >/dev/null; then
    echo "CRITICAL: model mismatch ${backend}" >&2
    failed=1
  fi
done
exit "${failed}"

Even a successful run does not guarantee capacity; continuous monitoring of 4xx/5xx rates, connection counts, request latency, upstream errors, model queue length, first‑token latency, token throughput, GPU utilization, and memory is required.

13. Release Acceptance and Rollback Criteria

Before going live, verify that all backends are equivalent, direct requests succeed, Nginx syntax is clean, SSE is not cached, POST retries are disabled, timeouts meet business needs, logs contain request IDs and upstream timings, rate‑limit keys cannot be forged, maintenance nodes can be drained safely, backups are restorable, and TLS private‑key permissions remain tight.

Rollback is considered successful only when gateway health, core request success, error rate, streaming output, request distribution, and absence of traffic to failed nodes are all restored. A successful nginx -t after reload is merely evidence of configuration validity, not business recovery.