Nginx Load Balancing for LLM Instances: Configuration, Pitfalls, and Best Practices

1. Determine What Can Be Load Balanced

The load balancer can only safely distribute traffic among "equivalent backends." If instances load different model weights, quantization precision, or chat templates, output behavior may differ even with HTTP 200. Before going live, verify the following items:

Model logical name : /v1/models must return consistent IDs; otherwise clients randomly see "model not found."

API structure : Status codes and SSE format must match; otherwise SDKs intermittently fail to parse.

Context and output limits : Must be identical within the same business pool; otherwise long requests randomly succeed or fail.

Chat template : Same version required; otherwise output style and capability drift.

GPU capacity : Explicitly confirm whether instances are homogeneous; equal-weight distribution may overload smaller instances.

Timeout strategy : Server, Nginx, and client timeouts must be coordinated; otherwise the middle layer disconnects prematurely.

First, connect directly to each backend bypassing Nginx. The script below checks TCP, health endpoint, and model list, explicitly listing failed nodes; /health existence depends on the actual inference framework.

#!/usr/bin/env bash
set -euo pipefail

BACKENDS=(
  "http://10.20.0.11:8000"
  "http://10.20.0.12:8000"
  "http://10.20.0.13:8000"
)

for backend in "${BACKENDS[@]}"; do
  echo "Checking ${backend}"
  curl -fsS --connect-timeout 2 --max-time 5 "${backend}/health" >/dev/null
  curl -fsS --connect-timeout 2 --max-time 10 "${backend}/v1/models" | jq -r '.data[].id'
done

Consistent model names are necessary but not sufficient; send the same low-randomness request to confirm response structure. Do not require identical output text because floating-point computation, parallel strategies, and sampling implementations can still cause differences.

for host in 10.20.0.11 10.20.0.12 10.20.0.13; do
  curl -fsS "http://${host}:8000/v1/chat/completions" \
    -H 'Content-Type: application/json' \
    -d '{
      "model":"<model_name>",
      "messages":[{"role":"user","content":"reply ready only"}],
      "temperature":0,
      "max_tokens":16,
      "stream":false
    }' | jq -c '{model,finish_reason:.choices[0].finish_reason,usage}'
done

2. Check Nginx Version and Compilation Capabilities

Different distributions package Nginx with different modules and default paths. Record version, compile options, and loaded main configuration before deployment, especially confirming presence of TLS, real IP, or status modules.

nginx -v
nginx -V 2>&1 | tr ' ' '
' | sort
sudo nginx -T > /tmp/nginx-effective-config.txt

nginx -T

outputs the full configuration, which may contain internal domains, certificate paths, or authentication info; treat the file as sensitive. Do not paste unsanitized output into public tickets.

3. Choose a Load Balancing Algorithm

Nginx open source commonly uses round-robin, least_conn, ip_hash, and hash with consistent. LLM request latency varies widely (2 seconds to minutes), so simple round-robin only counts requests and cannot reflect existing long requests. least_conn is usually a better baseline for homogeneous inference instances: new requests go to the node with the fewest active connections. least_conn still does not know token count, KV cache usage, or queue length. A single super-long context request may be heavier than many short ones, so it is a usable baseline, not an optimal scheduler. If session-level cache affinity is required, use consistent hashing on a trusted session header, but this reduces cache-hit stability during failover and may create hotspots.

The following configuration creates a homogeneous model pool. max_fails=2 fail_timeout=20s is passive failure detection: after the failure window reaches the threshold, the node becomes temporarily unavailable. Nginx open source does not provide built-in active health checks like the commercial version.

upstream llm_backend {
    least_conn;
    zone llm_backend 64k;

    server 10.20.0.11:8000 max_fails=2 fail_timeout=20s;
    server 10.20.0.12:8000 max_fails=2 fail_timeout=20s;
    server 10.20.0.13:8000 max_fails=2 fail_timeout=20s;

    keepalive 64;
}

zone

lets workers share upstream runtime state, suitable for multi-worker setups. keepalive 64 is the per-worker idle upstream connection cache limit, not a global concurrency limit or business request limit.

Heterogeneous instances can use weights, but weights should come from capacity tests under the same request profile, not simple VRAM size conversion. Example: strong instance weight 2, weak instance weight 1.

upstream llm_backend_weighted {
    least_conn;
    zone llm_backend_weighted 64k;

    server 10.20.0.21:8000 weight=2 max_fails=2 fail_timeout=20s;
    server 10.20.0.22:8000 weight=1 max_fails=2 fail_timeout=20s;
    keepalive 64;
}

Do not mix different models in the same pool and rely on weights to mask functional differences. Different models should go into separate upstreams, then routed by path, domain, or validated model field.

4. Write Proxy Configuration for Non-Streaming and Streaming APIs

OpenAI-compatible LLM interfaces typically use plain HTTP POST; streaming uses Server-Sent Events (SSE), not WebSocket. Key points: disable response buffering, set sufficient read timeouts, pass request IDs, and allow upstream keepalive reuse.

Core server configuration:

server {
  listen 443 ssl http2;
  server_name <API_domain>;

  ssl_certificate     <cert_path>;
  ssl_certificate_key <key_path>;

  client_max_body_size 10m;
  client_body_timeout 30s;

  location /v1/ {
    proxy_pass http://llm_backend;
    proxy_http_version 1.1;
    proxy_set_header Connection "";
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_set_header X-Request-ID $request_id;

    proxy_connect_timeout 3s;
    proxy_send_timeout 60s;
    proxy_read_timeout 600s;

    proxy_request_buffering off;
    proxy_buffering off;
    gzip off;

    proxy_next_upstream error timeout http_502 http_503 http_504;
    proxy_next_upstream_tries 2;
    proxy_next_upstream_timeout 5s;
  }
}

proxy_set_header Connection ""

clears client hop-by-hop headers to enable HTTP/1.1 upstream keepalive. proxy_buffering off ensures streaming data is forwarded immediately; proxy_request_buffering off avoids waiting for the entire request body before sending to the backend. proxy_read_timeout 600s is the idle time allowed between two upstream read operations, not the absolute total inference duration. Set it according to business max generation time and coordinate with client, API gateway, and inference service. Too short interrupts normal long generations; too long lets abnormal connections occupy resources longer.

Newer Nginx versions recommend standalone http2 on; directive; older versions use listen 443 ssl http2;. Verify syntax with nginx -v and current version docs; if changing syntax, run nginx -t again and do not keep conflicting listen configs.

Retry must be extremely cautious. Chat Completions uses POST; Nginx by default will not replay non-idempotent requests to the next node after the request has been sent upstream, unless non_idempotent is explicitly added. Do not add non_idempotent to improve "success rate": the model request may continue consuming GPU on the first node while the second node executes again, causing duplicate billing, double tool calls, or business side effects. The above retry mainly covers pre-connection errors and safely switchable cases.

If the inference framework sets X-Accel-Buffering: no, keep it; but the gateway should still explicitly disable buffering to avoid inconsistent backend behavior. Do not enable gzip for SSE; compression may add buffering and increase first-token latency.

5. Separate Health Checks from Business Traffic

Health checks should not go through the generic /v1/ load balancing path and be answered by an arbitrary node; that only proves "at least one node is alive." Operational checks should directly connect to each backend; user-facing health can return Nginx's own status or proxy to a lightweight model list endpoint.

Add a local health endpoint at the load balancing entry that does not trigger inference:

location = /gateway-health {
  access_log off;
  default_type text/plain;
  return 200 "ok
";
}

This endpoint only proves the Nginx worker can respond, not that the backend model is available. Monitoring systems should retain three layers: gateway health, per-instance health, and periodic low-frequency synthetic requests, to avoid single-probe misjudgment.

6. Record Access Logs That Form an Evidence Chain

Default combined logs lack upstream address, upstream latency, and total request latency, making it impossible to answer "which node did the request hit, and where is the slowness?" Use JSON logs capturing request ID, status, upstream node, and timings. Do not log request bodies, Authorization headers, or model outputs.

log_format llm_json escape=json
'{
  "time":"$time_iso8601",
  "request_id":"$request_id",
  "remote_addr":"$remote_addr",
  "method":"$request_method",
  "uri":"$uri",
  "status":$status,
  "request_length":$request_length,
  "bytes_sent":$bytes_sent,
  "request_time":$request_time,
  "upstream_addr":"$upstream_addr",
  "upstream_status":"$upstream_status",
  "upstream_connect_time":"$upstream_connect_time",
  "upstream_header_time":"$upstream_header_time",
  "upstream_response_time":"$upstream_response_time"
}';

access_log /var/log/nginx/llm_access.json llm_json;
error_log  /var/log/nginx/llm_error.log warn;

$upstream_header_time

approximates first response header arrival but is not equivalent to model first-token latency; SSE first token requires client-side or observability component recording at the data frame level. If upstream retry occurs, some upstream variables contain comma-separated multi-values, which is evidence of node switching.

Use jq to summarize recent log node distribution and status codes (read-only, no service modification):

sudo tail -n 5000 /var/log/nginx/llm_access.json \
  | jq -r '[.upstream_addr, (.status|tostring)] | @tsv' \
  | sort | uniq -c | sort -nr

Identify requests with high total latency and high upstream latency, then correlate with backend logs using request ID:

sudo jq -c 'select((.request_time // 0) > 30) |
  {time,request_id,status,request_time,upstream_addr,upstream_response_time}' \
  /var/log/nginx/llm_access.json | tail -n 100

If request_time is high but upstream_response_time is low, suspect slow client read, slow request body upload, or log field parsing issues; if both are high, examine model queue, input/output tokens, and GPU metrics. Root cause must be supported by multiple evidences.

7. Rate Limiting, Connection Limits, and Overload Protection

Nginx's limit_req counts requests, not token cost. It suits burst call limiting but cannot replace business rate limiting based on tokens, user quotas, and model queue length. When rate limiting by API Key, never log the plaintext key; if the auth layer maps callers to internal tenant IDs, partition by a trusted request header.

map $http_x_tenant_id $tenant_key {
  default $http_x_tenant_id;
  ""      $binary_remote_addr;
}

limit_req_zone $tenant_key zone=llm_per_tenant:20m rate=2r/s;
limit_conn_zone $tenant_key zone=llm_conn_per_tenant:20m;

server {
  location /v1/ {
    limit_req zone=llm_per_tenant burst=10 nodelay;
    limit_conn llm_conn_per_tenant 8;

    proxy_pass http://llm_backend;
    # other proxy params same as main config
  }
}

Only use X-Tenant-ID as a rate limit key when it is injected by a trusted auth gateway and cannot be forged by external clients. Trusting client-supplied tenant headers lets callers bypass limits easily. Connection limits must also consider reasonable parallel streaming needs per user.

Set clear status codes for rate limiting so clients can distinguish overload from service failure:

limit_req_status 429;
limit_conn_status 429;

Clients receiving 429 should exponential backoff with jitter; immediate unlimited retries amplify temporary overload into a retry storm.

8. Applicability Boundaries of Session-Based Consistent Hashing

If the backend enables prefix caching and many multi-turn requests share a stable common prefix, consistent hashing can increase the probability of the same session returning to the same instance. First, the application generates an unguessable, stable, length-limited X-Session-ID, then configure:

upstream llm_session_backend {
  zone llm_session_backend 64k;
  hash $http_x_session_id consistent;

  server 10.20.0.11:8000 max_fails=2 fail_timeout=20s;
  server 10.20.0.12:8000 max_fails=2 fail_timeout=20s;
  server 10.20.0.13:8000 max_fails=2 fail_timeout=20s;
  keepalive 64;
}

Consistent hashing does not synchronize KV cache; node restart or scaling may route requests to a new node requiring recomputation. A single large tenant may create a hotspot. Therefore, compare cache hit rate, load skew, and failure recovery rather than enabling stickiness simply because multi-turn dialogue exists.

9. Configuration Changes Must Follow a Complete Closed Loop

Directly editing production config and restarting is high risk. Nginx supports graceful reload; adopt "backup, generate new file, syntax check, reload, verify, rollback" sequence. Reload lets new workers use new config while old workers finish existing connections, protecting long streaming requests.

Script to backup main config directory and run syntax check; backup directory must be on controlled storage, not copying private keys to unprotected locations.

#!/usr/bin/env bash
set -euo pipefail

NGINX_DIR="/etc/nginx"
BACKUP_ROOT="<config_backup_dir>"
STAMP="$(date +%Y%m%d-%H%M%S)"
BACKUP_DIR="${BACKUP_ROOT}/nginx-${STAMP}"

sudo install -d -m 0700 "${BACKUP_DIR}"
sudo cp -a "${NGINX_DIR}/." "${BACKUP_DIR}/"
sudo nginx -t
echo "Backup created: ${BACKUP_DIR}"

After modifications, view diff, then test and reload. systemctl reload must not run if nginx -t fails.

sudo diff -u <old_config> <new_config> || true
sudo nginx -t
sudo systemctl reload nginx
sudo systemctl status nginx --no-pager

Immediately verify local entry, TLS, model list, non-streaming and streaming requests. --resolve can temporarily resolve the domain to a specific gateway IP before DNS cutover, preserving correct Host and TLS SNI.

curl --fail --silent --show-error \
  --resolve '<API_domain>:443:<gateway_IP>' \
  https://<API_domain>/gateway-health

curl --fail --silent --show-error \
  --resolve '<API_domain>:443:<gateway_IP>' \
  https://<API_domain>/v1/models | jq .

If errors appear, rollback config and re-run syntax check. Rollback changes routing for new requests, but established streaming connections may continue on old workers; observe worker and connection drain, do not force-kill immediately.

sudo rsync -a --delete <config_backup_dir>/nginx-<timestamp>/ /etc/nginx/
sudo nginx -t
sudo systemctl reload nginx
curl -fsS https://<API_domain>/gateway-health

rsync --delete

removes files in target not present in source; high-risk operation. Confirm source and target are exact directories, first dry-run with rsync -ani --delete; if only one file changed, safer to restore just that file.

10. Instance Maintenance and Connection Draining

Before taking a backend down, confirm target node and impact scope. Safest method: mark the server down or temporarily remove from upstream, syntax check and reload; new requests stop going to that node, existing connections are handled by old workers until completion.

Maintenance upstream example:

upstream llm_backend {
    least_conn;
    zone llm_backend 64k;

    server 10.20.0.11:8000 max_fails=2 fail_timeout=20s;
    server 10.20.0.12:8000 down;
    server 10.20.0.13:8000 max_fails=2 fail_timeout=20s;
    keepalive 64;
}

After reload, confirm from access logs that no new requests are assigned to the maintenance node. Do not rely on a single curl; observe a window covering the longest common request.

sudo nginx -t && sudo systemctl reload nginx
sudo tail -F /var/log/nginx/llm_access.json \
  | jq --unbuffered 'select(.upstream_addr | contains("10.20.0.12:8000"))'

Once new requests stop and in-flight requests drain, stop the model instance. Stopping interrupts unfinished requests, so do it during low traffic, with impact notification and quick recovery commands ready.

After maintenance, directly verify the node, then remove down and gradually restore. Open source Nginx static config cannot do percentage-based slow start; give the restored node a lower weight first, observe error rate and latency, then restore normal config.

11. Common Failure Diagnosis

1. 502 Bad Gateway

502 means Nginx received no valid response from upstream; common causes: port not listening, connection refused, process exited, invalid response headers. Verify routing and TCP from gateway host, then directly hit health endpoint.

ip route get 10.20.0.11
nc -vz -w 2 10.20.0.11 8000
curl -v --connect-timeout 2 --max-time 10 \
  http://10.20.0.11:8000/health
sudo grep -F '10.20.0.11:8000' /var/log/nginx/llm_error.log | tail -n 50

Only if logs show connect() failed (111: Connection refused) and direct port connection fails can you confirm connection refusal; do not attribute all 502s to "backend down."

2. 504 Gateway Timeout

504 usually relates to waiting for upstream response timeout. Correlate Nginx and backend logs using request ID, then check for long queue, long generation, or GPU anomaly. Blindly increasing proxy_read_timeout only masks the problem.

REQUEST_ID="<request_ID>"

sudo grep -F "${REQUEST_ID}" /var/log/nginx/llm_access.json
sudo grep -F "${REQUEST_ID}" /var/log/nginx/llm_error.log
ssh <backend_host> "journalctl -u <model_service> --since '-15 min' --no-pager | grep -F '${REQUEST_ID}'"

Remote command requires controlled SSH permissions on target. Production environments should prefer centralized logging platforms to avoid ops accounts having unnecessary permissions on multiple inference nodes.

3. Streamed Response Returned All at Once

First, directly hit backend with curl -N, then compare via Nginx. If direct connection streams chunk-by-chunk but proxy returns once, focus on buffering, compression, and CDN policies at all intermediate layers.

curl -N -sS http://10.20.0.11:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"<model_name>","messages":[{"role":"user","content":"list five checks"}],"stream":true,"max_tokens":128}'

curl -N -sS https://<API_domain>/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"<model_name>","messages":[{"role":"user","content":"list five checks"}],"stream":true,"max_tokens":128}'

4. Severe Request Distribution Imbalance

First, from logs, count requests and total processing time per node. Uneven request count alone is not necessarily a fault: least_conn adjusts based on active connections; node processing speed, request length, and failed retries all affect distribution.

sudo jq -r 'select(.upstream_addr != "") |
  [.upstream_addr, (.request_time|tostring)] | @tsv' \
  /var/log/nginx/llm_access.json \
  | awk -F '\t' '{count[$1]++; sum[$1]+=$2}
    END {for (h in count) printf "%s requests=%d total_time=%.3f
", h, count[h], sum[h]}'

If a node has few requests but many connection failures, check service health; if few requests but total processing time similar to others, it may be handling longer requests; if using session hashing, check for tenant or session hotspots.

12. Automated Multi-Node Inspection

The script below checks config syntax, gateway health, per-backend health, and model name consistency. It does not auto-reload or remove nodes because automatic routing changes may amplify misjudgment impact; anomalies are handled by monitoring alerts and change processes.

#!/usr/bin/env bash
set -euo pipefail

EXPECTED_MODEL="${EXPECTED_MODEL:-<model_name>}"
GATEWAY_URL="${GATEWAY_URL:-https://<API_domain>}"
BACKENDS=(
  "http://10.20.0.11:8000"
  "http://10.20.0.12:8000"
  "http://10.20.0.13:8000"
)

sudo nginx -t >/dev/null
curl -fsS --max-time 5 "${GATEWAY_URL}/gateway-health" >/dev/null

failed=0
for backend in "${BACKENDS[@]}"; do
  if ! curl -fsS --max-time 5 "${backend}/health" >/dev/null; then
    echo "CRITICAL: health failed ${backend}" >&2
    failed=1
    continue
  fi
  if ! curl -fsS --max-time 10 "${backend}/v1/models" \
      | jq -e --arg m "${EXPECTED_MODEL}" '.data[] | select(.id == $m)' >/dev/null; then
    echo "CRITICAL: model mismatch ${backend}" >&2
    failed=1
  fi
done

exit "${failed}"

Successful inspection does not guarantee capacity headroom. Also monitor Nginx 4xx/5xx, connection counts, request latency, upstream errors, model queue, first-token latency, output token throughput, GPU utilization and VRAM. Only by placing gateway and inference metrics on the same timeline can you distinguish entry faults, network faults, and model overload.

13. Production Acceptance and Rollback Standards

Pre-launch verification must cover: all backend models and configs equivalent; direct requests pass; Nginx config syntax correct; SSE not cached; POST non-idempotent retry disabled; timeouts cover business needs but are bounded; access logs contain request ID, upstream, and latency; rate limit keys cannot be forged by clients; maintenance nodes can be drained before stop; old config backup restorable; TLS private key permissions not widened by backup.

Rollback criteria must be quantified: gateway health restored, core requests succeed, error rate drops, streaming output normal, requests redistributed to old pool, no continued traffic to failed nodes. Nginx reload success is only process evidence, not business recovery conclusion.

For LLM services, load balancing is not "evenly distribute every request" but maintaining availability across inference tasks of varying lengths and costs. Nginx provides a reliable entry and basic scheduling, but token cost, queue depth, prefix caching, and tenant quotas still require joint governance by the inference platform and business layer.