Nginx Load Balancing: Production-Grade Upstream Configuration & Traffic Control

1. Validate Runtime Environment & Current Configuration

Before modifying upstream, confirm Nginx version, compiled modules, systemd unit, main config path, and include hierarchy. Many production issues stem from editing non-included files, reloading on host instead of container, or mixing distro directory layouts.

1.1 Check Version, Compile Options & Key Modules

nginx -V

Output goes to stderr. Verify version, --conf-path, --prefix, --with-http_stub_status_module. Open source passive failure removal differs from Plus active health checks; do not apply Plus directives to open source.

1.2 Confirm systemd Unit Managing Nginx

systemctl status nginx --no-pager -l
systemctl cat nginx

First shows status, main PID, recent logs; second reveals ExecStart, PID file, extra env vars. If Nginx runs in container/Kubernetes/other process manager, do not use host systemctl for deploy/rollback; operate at actual runtime location.

1.3 Query Config Paths & Possible Site Include Directories

nginx -V 2>&1 | tr ' ' '
' | grep -E '^--(conf-path|prefix|sbin-path)='
sudo find /etc/nginx -maxdepth 3 -type f -name '*.conf' -print

First extracts paths from compile args; second lists candidate files but cannot confirm each is active. Debian/Ubuntu use sites-available/sites-enabled; RHEL uses conf.d. Final authority is nginx -T include output.

1.4 Syntax Check Before Any Change

sudo nginx -t

Checks syntax and attempts to open referenced files. Success does not guarantee backend reachable, DNS resolvable, or business path correct, but failure means never reload. Establishes baseline that original config is valid.

1.5 Export Full Parsed Config for Audit

sudo nginx -T > /tmp/nginx-effective-<site-name>.conf 2>&1
sudo grep -nE 'upstream |server_name |proxy_pass ' /tmp/nginx-effective-<site-name>.conf

nginx -T

dumps main + includes to stdout. Redirect to file (may contain internal domains, cert paths, ACLs) with restricted permissions and clean promptly. Confirms new file is included, detects duplicate upstream definitions. Do not commit temp files to public repos.

1.6 Verify Each Backend Responds Directly

curl --fail --silent --show-error --max-time 3 
  http://<backend-ip1>:<backend-port><health-check-path>
curl --fail --silent --show-error --max-time 3 
  http://<backend-ip2>:<backend-port><health-check-path>

Validates direct reachability and health-check path per backend. Path must be read-only, no side effects, lightweight; avoid URLs that create sessions or trigger order logic. If direct access fails, fix backend service, network policy, or listen address first; Nginx reload cannot solve backend unavailability.

1.7 Check Backend Listen Address, Port & Owning Process

sudo ss -ltnp | grep -E ':(<backend-port>)\b'

Run on backend host to confirm business actually listens on expected TCP port. If service binds only 127.0.0.1 while Nginx is on another machine, reverse proxy will fail; loopback only works when co-located. Network-layer evidence; still needs HTTP health check for application layer.

1.8 If Using Hostnames in Upstream, Verify DNS Resolution at Deploy Time

getent ahostsv4 <backend-hostname>
getent ahostsv6 <backend-hostname>

Open source Nginx resolves static hostnames at start/reload; resolution differs from variable usage. getent uses system NSS chain, confirming resolvability in that runtime. Do not treat temporary /etc/hosts bindings as long-term; they mask DNS changes and multi-replica scaling.

2. Build Minimal Usable Upstream & Forward Request Info Correctly

Open source upstream defaults to weighted round-robin. Define upstream in http context; server/location references via proxy_pass. Explicitly set timeouts, Host, and forwarding headers to avoid backend logs identifying all traffic as Nginx itself, or default timeouts causing connection pile-up.

2.1 Define Basic Backend Pool with Weighted Round-Robin

upstream <site-name>_backend {
  server <backend-ip1>:<backend-port> weight=3;
  server <backend-ip2>:<backend-port> weight=1;
  keepalive 32;
}

Weight 3 vs 1 means former gets ~3x new requests when both healthy, not guaranteed 3x throughput. keepalive reuses idle Nginx-to-backend connections; size must be tuned via load test against worker count, concurrency, backend connection capacity. It is not client keepalive. Omit weight when specs identical and no bias needed.

2.2 Configure Reverse Proxy Headers & Timeouts in location

location / {
  proxy_pass http://<site-name>_backend;
  proxy_http_version 1.1;
  proxy_set_header Host $host;
  proxy_set_header X-Real-IP $remote_addr;
  proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
  proxy_set_header X-Forwarded-Proto $scheme;
  proxy_connect_timeout 3s;
  proxy_read_timeout 30s;
  proxy_send_timeout 30s;
}

Host

preserves user-accessed domain; X-Forwarded-For appends proxy chain, not simply $remote_addr which would overwrite existing chain. Three timeouts cover connect to backend, read backend response, send request to backend; they are not total request latency. 30s is example; must align with app SLA, long connections, uploads, gateway timeouts.

2.3 Define Access Log Format with Upstream State Fields

log_format upstream_timing
  '$remote_addr $host "$request" $status '
  'request_time=$request_time '
  'upstream_addr=$upstream_addr '
  'upstream_status=$upstream_status '
  'upstream_connect_time=$upstream_connect_time '
  'upstream_header_time=$upstream_header_time '
  'upstream_response_time=$upstream_response_time';

Provides total request time plus backend address, status, connect time, first-byte time, full response time — key to deciding "slow in Nginx or slow in backend". Variables may contain multiple comma/colon-separated values indicating retries or internal redirects; do not treat as single backend. Access logs contain client data; follow masking/retention policies.

2.4 Complete Site Config: Upstream, Logs & HTTPS Entry

upstream <site-name>_backend {
  server <backend-ip1>:<backend-port> max_fails=3 fail_timeout=10s;
  server <backend-ip2>:<backend-port> max_fails=3 fail_timeout=10s;
  keepalive 32;
}

server {
  listen 443 ssl;
  server_name <domain>;
  access_log /var/log/nginx/<site-name>-access.log upstream_timing;
  error_log  /var/log/nginx/<site-name>-error.log warn;
  location / {
    proxy_pass http://<site-name>_backend;
    proxy_http_version 1.1;
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_connect_timeout 3s;
    proxy_read_timeout 30s;
  }
}

Example fits existing http include tree; SSL cert/key omitted due to environment variance — cannot be missing. max_fails / fail_timeout are passive failure detection: within fail_timeout window, reaching failure count marks node temporarily unavailable; not equivalent to active health checks. Before applying, ensure no existing server block conflicts on same listen+server_name.

2.5 Controlled Deploy Script: Backup, Write, Test, Reload

#!/usr/bin/env bash
set -euo pipefail
CONF_FILE="<config-file>"
BACKUP_DIR="/var/backups/nginx"
STAMP="$(date -u +%Y%m%dT%H%M%SZ)"
sudo install -d -m 0750 "${BACKUP_DIR}"
sudo cp -a "${CONF_FILE}" "${BACKUP_DIR}/$(basename "${CONF_FILE}").${STAMP}"
sudo nginx -t
sudo systemctl reload nginx
sudo systemctl is-active --quiet nginx
printf 'backup=%s
' "${BACKUP_DIR}/$(basename "${CONF_FILE}").${STAMP}"

Assumes new config already safely written to <config-file>; does not replace code review or config generation. Backup first, then nginx -t; syntax failure prevents reload. Reload loads smoothly, old workers finish existing connections. Verify sudo rights, disk space, Nginx service name; containerized envs should use image/orchestrator deploy methods.

2.6 Independent Syntax Check & Smooth Reload

sudo nginx -t && sudo systemctl reload nginx
sudo systemctl status nginx --no-pager -l

Most common config-activation combo. Do not habitually replace reload with restart: restart drops existing connections, raises deploy risk. Syntax pass does not guarantee routing hits correct upstream; post-reload verify actual distribution via requests, logs, monitoring.

3. Verify Traffic Actually Distributes to Expected Backends

Validation must cover entry availability, backend distribution, error rates. Mere 200 OK does not prove all nodes healthy — requests may always hit same backend; Nginx process healthy does not mean app returns expected content.

3.1 Basic Health Check from Entry (Fail-Fast)

curl --fail --silent --show-error --location 
  --connect-timeout 3 --max-time 10 
  https://<domain><health-check-path>

--fail

makes HTTP 400+ return non-zero, suitable for automated gates; --location follows explicit redirects. If health check needs Host routing but entry only has IP, use --resolve <domain>:443:<entry-ip> to test explicitly, avoiding /etc/hosts edits. Request must originate from permitted probe network.

3.2 Sequential Requests Recording Backend Identifier from Response Headers

for i in $(seq 1 20); do
  curl --silent --show-error --fail \
    -H 'Connection: close' \
    -D - -o /dev/null \
    "https://<domain><health-check-path>" \
    | grep -i '^X-Backend-Id:'
done

Requires backend to return X-Backend-Id on non-sensitive health endpoint to verify multi-instance coverage. Connection: close forces new client connection each iteration, preventing connection reuse from masking distribution; not a production performance setting. If app lacks this header, use upstream access logs or controlled tracing — never guess traffic spread.

3.3 When Session Stickiness Required, Use ip_hash Not Default Round-Robin

upstream <site-name>_sticky_backend {
  ip_hash;
  server <backend-ip1>:<backend-port>;
  server <backend-ip2>:<backend-port>;
}

ip_hash

selects backend by client address, reducing session drift in legacy systems without shared session store. However, NAT/proxy behind many users may concentrate on one backend; IPv6 and proxy chains alter effect. Prefer evaluating shared session storage or app-layer consistency; do not enable stickiness unconditionally for "more stable load balancing".

3.4 When Request Latency Varies Widely, Use least_conn for New Connections

upstream <site-name>_least_conn_backend {
  least_conn;
  server <backend-ip1>:<backend-port>;
  server <backend-ip2>:<backend-port>;
}

least_conn

sends new requests to server with fewest active connections, suitable for highly variable processing times. It observes Nginx-side connection count, not backend CPU, queue, or DB pressure; superiority over round-robin must be proven with real metrics. When combined with ip_hash etc., confirm Nginx version and directive compatibility.

3.5 When Stable Mapping Needed with Node Changes Allowed, Use Consistent Hash

upstream <site-name>_hash_backend {
  hash $request_uri consistent;
  server <backend-ip1>:<backend-port>;
  server <backend-ip2>:<backend-port>;
  server <backend-ip3>:<backend-port>;
}

consistent

minimizes key remapping when backends added/removed, good for cache sharding, stateless reads wanting stable hits. $request_uri includes query params; random params can scatter hotspots or invalidate cache. Hash key must be designed per business semantics. Consistent hash cannot replace backend capacity planning and failure handling.

4. Failover Boundaries: Open Source Nginx Relies Primarily on Passive Failure

Open source Nginx should not be described as having general-purpose active HTTP health checks. Standard upstream max_fails, fail_timeout, backup, down rely on observed connection/response failures during real proxy requests. For active probing needs, implement in external health checks, service discovery, or load balancer layer, and update Nginx config via controlled deploy.

4.1 Configure Passive Failure Thresholds & Backup Node

upstream <site-name>_backend {
  server <backend-ip1>:<backend-port> max_fails=3 fail_timeout=10s;
  server <backend-ip2>:<backend-port> max_fails=3 fail_timeout=10s;
  server <backup-backend-ip>:<backend-port> backup;
}

When regular nodes accumulate enough failures within fail_timeout, Nginx temporarily avoids them; backup node participates only when all regular nodes unavailable. Failure counting interacts with proxy_next_upstream; test connect-refused, timeout, 5xx behavior before go-live. Backup node should not carry main traffic long-term; monitor its capacity and version separately.

4.2 Explicitly Define Which Upstream Responses Trigger Next Backend Attempt

location / {
  proxy_pass http://<site-name>_backend;
  proxy_next_upstream error timeout http_502 http_503 http_504;
  proxy_next_upstream_tries 2;
  proxy_next_upstream_timeout 5s;
}

Only for idempotent or retry-designed requests. Blindly retrying POST, payments, writes may cause duplicate submissions; business must use idempotency keys or split such paths into non-retry locations. tries and timeout limit retry amplification; total latency must fit caller timeout budget.

4.3 When Maintaining a Backend, Explicitly Mark down Then Reload

upstream <site-name>_backend {
  server <backend-ip1>:<backend-port>;
  server <backend-ip2>:<backend-port> down;
  server <backend-ip3>:<backend-port>;
}

down

excludes node from selection, suitable for pre-maintenance drain. High-risk point: can remaining instances absorb traffic? Check connections, CPU, queue, error rate before drain; canary one instance first. After config reload, verify node no longer appears in $upstream_addr. On maintenance end, remove down, test, reload, gradually restore traffic.

4.4 Extract Direct Evidence of Upstream Connection Failures from Error Log

sudo grep -E 'connect\(\) failed|upstream timed out|no live upstreams' \
  /var/log/nginx/<site-name>-error.log \
  | tail -n 100

connect() failed

often means connection refused or network unreachable; upstream timed out indicates wait timeout; no live upstreams means Nginx thinks no nodes available. Each requires different handling; must correlate with backend listen, network path, access log upstream_status, and deploy records — never attribute all 502 to Nginx.

4.5 Check TCP Connection State from Nginx to Backend for Abnormal Pile-Up

sudo ss -tanp '( dport = :<backend-port> )' \
  | awk 'NR>1 {state[$1]++} END {for (s in state) print s, state[s]}'

Run on Nginx node to count TCP states to backend port. Many SYN-SENT suggests connection establishment issues; many ESTAB not necessarily abnormal — correlate with concurrency and keepalive config; TIME-WAIT may be normal short-connection mode. Auxiliary evidence; cannot alone conclude "backend saturated".

5. Observability: Record Upstream Selection & Latency

Without upstream address, status, and timing in logs, troubleshooting stalls at "entry 5xx up". Combine log format, status page, external monitoring, alerting; but note open source stub_status only gives aggregate connections/requests, no per-upstream detail.

5.1 Expose Open Source stub_status on Restricted Address

server {
  listen 127.0.0.1:8080;
  server_name localhost;
  location = /nginx_status {
    stub_status;
    allow 127.0.0.1;
    deny all;
  }
}

Requires http_stub_status_module compiled in. Bind loopback and restrict access to avoid exposing connection counts and request rates to public; if Prometheus exporter not co-located, use controlled network, auth proxy, or dedicated collection channel. Cannot replace access log upstream details.

5.2 Query stub_status & Confirm Status Page Not Publicly Accessible

curl --fail --silent --show-error http://127.0.0.1:8080/nginx_status
curl --fail --silent --show-error --max-time 3 \
  http://<entry-ip>:8080/nginx_status && exit 1 || true

First validates local collection path; second expects non-local entry access to fail. Second uses && exit 1 || true so "unexpectedly accessible" triggers handling. Different firewalls/bindings yield different errors; acceptance criterion is not specific text but that public/unauthorized segments cannot read status page.

5.3 Analyze Slowest Backend Responses from Structured Fields

sudo awk '
  match($0, /upstream_addr=([^ ]+)/, a) &&
  match($0, /upstream_response_time=([^ ]+)/, t) {
    print a[1], t[1]
  }
' /var/log/nginx/<site-name>-access.log \
  | sort -k2,2nr \
  | head -n 50

Awk depends on key-value log format from step 11. upstream_response_time may contain comma-separated retry chains; complex cases better parsed after shipping to log platform. Sorted results are slow-request candidates; must correlate by backend, path, status code, time window, deploy version — never judge node failure from single top-50.

5.4 Periodic Entry Health Check via systemd Timer

# /etc/systemd/system/<site-name>-proxy-health.service
[Unit]
Description=Check Nginx upstream entry health for <site-name>

[Service]
Type=oneshot
ExecStart=/usr/local/sbin/<site-name>-proxy-health.sh

# /etc/systemd/system/<site-name>-proxy-health.timer
[Unit]
Description=Run Nginx proxy health check every minute

[Timer]
OnBootSec=2min
OnUnitActiveSec=1min
Persistent=true

[Install]
WantedBy=timers.target

Timer for lightweight entry validation; not a substitute for centralized monitoring, synthetic transactions, or app metrics. Frequency and timeout per service scale to avoid probe burden. After saving units, daemon-reload and verify run account has script exec and network access rights.

5.5 Read-Only, Fail-Non-Zero Entry Probe Script

#!/usr/bin/env bash
set -euo pipefail
URL="https://<domain><health-check-path>"
MAX_TIME=5
status="$(curl --silent --show-error --output /dev/null \
  --write-out '%{http_code}' \
  --connect-timeout 2 --max-time ${MAX_TIME} \
  ${URL})"
test "${status}" = "200"
printf 'url=%s status=%s
' "${URL}" "${status}"

Script strictly accepts 200; if business health endpoint returns 204 or needs Host, cert chain, mTLS, adjust per actual protocol — do not disable TLS verification to make check pass. systemd logs non-zero exit; monitoring can alert on it. Script does not auto-reload, switch backends, or restart services, avoiding probe-triggered fault amplification.

5.6 Enable Probe Timer & Review Recent Failures

sudo chmod 0755 /usr/local/sbin/<site-name>-proxy-health.sh
sudo systemctl daemon-reload
sudo systemctl enable --now <site-name>-proxy-health.timer
systemctl list-timers <site-name>-proxy-health.timer --all
sudo journalctl -u <site-name>-proxy-health.service -n 50 --no-pager

Confirm script path, unit filename, permissions match. Last two verify timer scheduled and recent results. If site requires specific Host or goes through CDN, check URL must be at network position representing real user path; loopback success ≠ public entry healthy.

6. Canary Deploy, Scale & Rollback

Changing upstream is traffic mutation, not ordinary file edit. Operate on few entry nodes or one failure domain at a time; observe connections, error rate, P95/P99, backend resources, $upstream_addr distribution before proceeding. Adding new nodes to all entries, switching algorithm, and adjusting timeouts simultaneously makes root cause indistinguishable.

6.1 Pre-Check & Single-Node Canary for New Backend

curl --fail --silent --show-error --max-time 3 \
  http://<new-backend-ip>:<backend-port><health-check-path>
sudo nginx -t
sudo systemctl reload nginx
sudo tail -n 100 /var/log/nginx/<site-name>-access.log

First validates new backend from entry node; next three run only after new node added to one canary entry's config. Before execution, confirm that entry can be pulled from LB layer or tolerate limited blast radius. Watch logs for new node appearance, failure status, abnormal latency; no data ≠ success — verify canary entry actually receives traffic.

6.2 When Using DNS Names with resolver for Dynamic Resolution, Explicitly Set resolver

resolver <dns-server-ip> valid=30s;
resolver_timeout 2s;

set $backend_url http://<backend-hostname>:<backend-port>;
location / {
  proxy_pass $backend_url;
  proxy_set_header Host $host;
}

Variable in proxy_pass triggers resolver requirement; behavior differs from static upstream name resolution. DNS server must be controlled, available, returning correct records; valid=30s is cache TTL, not absolute real-time. Dynamic resolution changes failure model and debugging; validate DNS changes, TTL, IPv4/IPv6, backend app compatibility first. If no dynamic update needed, static IPs or controlled reloads are often easier to audit.

6.3 Restore Specific Config Backup Version & Smooth Rollback

#!/usr/bin/env bash
set -euo pipefail
CONF_FILE="<config-file>"
BACKUP_FILE="/var/backups/nginx/<config-filename>.<utc-timestamp>"
sudo test -r "${BACKUP_FILE}"
sudo cp -a "${BACKUP_FILE}" "${CONF_FILE}"
sudo nginx -t
sudo systemctl reload nginx
curl --fail --silent --show-error --max-time 10 \
  "https://<domain><health-check-path>"

Replace <config-filename> and <utc-timestamp> with actual backup name recorded in step 13. Rollback: verify backup readable, overwrite, syntax check, reload, then entry health check. Only restores this config file; if same change also adjusted DNS, firewall, certs, backend version, external LB, revert each per change manifest and record final state.

6.4 Post-Rollback Confirm No Residual New Upstream Addresses

sudo nginx -T 2>&1 | grep -F '<new-backend-ip>' && exit 1 || true
sudo grep -F '<new-backend-ip>' /var/log/nginx/<site-name>-access.log \
  | tail -n 20

First checks current effective config for lingering new node; second observes log for historical or delayed writes of that node. Log presence of old records does not mean rollback failed — correlate timestamps and entry requests. If first still matches, locate unrolled config in include chain rather than blindly reloading again.

7. Common Misconceptions

See 502 → restart Nginx. Root cause may be backend not listening, network reject, TLS/protocol mismatch, timeout, no live upstreams. First collect error log, access log upstream fields, backend listen evidence.

Treat max_fails as active health check. Open source learns failures from real proxy requests; low-traffic services may detect bad node very late.

Enable proxy_next_upstream on all endpoints. Non-idempotent write retries risk duplicate writes; must split location and retry policy by business semantics.

Syntax pass = deploy done. nginx -t only proves config parsable; must continue verifying entry, actual backend distribution, error rate, latency.

Expose stub_status for easier debugging. Status page is runtime info; default to minimal exposure with access control and audit.

8. Acceptance Criteria

After an upstream load balancing change, you should answer: Which config is currently effective? Which backends are actually reachable? What algorithm is used and why? How are connection failures, timeouts, 5xx handled? Which backends receive entry requests, with what latency and status? How is a maintenance node drained? How to recover to a specific backup version on failure? Answer with config, tests, logs, and metrics together to turn upstream from static snippet into operable traffic control component.