How to Safely Distribute Traffic with Nginx upstream Load Balancing

1. Verify Runtime and Current Configuration

Before touching upstream, confirm the Nginx version, compiled modules, systemd unit, main config path, and include hierarchy. Many production issues stem from editing files that are not included or reloading the wrong host. All checks below are read‑only.

1.1 Check version and compile options

nginx -V

Look for --conf-path, --prefix and --with-http_stub_status_module. Open‑source Nginx lacks the Plus health‑check directives, so do not copy Plus examples.

1.2 Identify the systemd unit managing Nginx

systemctl status nginx --no-pager -l
systemctl cat nginx

If Nginx runs inside a container or Kubernetes, use the actual runtime’s reload mechanism instead of the host’s systemctl.

1.3 Locate the effective configuration files

nginx -V 2>&1 | tr ' ' '
' | grep -E '^--(conf-path|prefix|sbin-path)='
sudo find /etc/nginx -maxdepth 3 -type f -name '*.conf'

Only the output of nginx -T (which prints the full include tree) is authoritative.

1.4 Perform a syntax check

sudo nginx -t

The command validates syntax and attempts to open referenced files. A successful run does not guarantee backend reachability, but any failure must block a reload.

1.5 Export the effective configuration for audit

sudo nginx -T > /tmp/nginx-effective-<site>.conf
sudo grep -nE 'upstream |server_name |proxy_pass ' /tmp/nginx-effective-<site>.conf

Store the dump with restricted permissions; it proves which upstream definitions are active.

1.6 Verify each backend can respond directly

curl --fail --silent --show-error --max-time 3 http://<backendIP1>:<port><healthPath>
curl --fail --silent --show-error --max-time 3 http://<backendIP2>:<port><healthPath>

The health path must be read‑only and side‑effect free. If a direct request fails, fix the backend service or network before reloading Nginx.

1.7 Check backend listening address and process

sudo ss -ltnp | grep -E ':<port>\b'

Run this on the backend host. If the service binds only to 127.0.0.1 while Nginx is on another machine, the proxy will always fail.

1.8 Verify DNS resolution for host‑named backends

getent ahostsv4 <backendDomain>
getent ahostsv6 <backendDomain>

Nginx resolves static hostnames at start‑up or reload; dynamic variables use the resolver directive.

2. Build a Minimal Working upstream and Pass Correct Headers

The open‑source default is weighted round‑robin. Define the upstream in the http context and reference it from a location block.

2.1 Define a weighted pool

upstream <site>_backend {
    server <backendIP1>:<port> weight=3;
    server <backendIP2>:<port> weight=1;
    keepalive 32;
}

Weight 3 vs 1 gives roughly a 3:1 request split when both nodes are healthy. Adjust keepalive based on worker count and connection capacity.

2.2 Configure proxy headers and timeouts

location / {
    proxy_pass http://<site>_backend;
    proxy_http_version 1.1;
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_connect_timeout 3s;
    proxy_read_timeout 30s;
    proxy_send_timeout 30s;
}

Preserve the original host, forward the client IP chain, and set explicit timeouts that match the service‑level agreement.

2.3 Log upstream timing for root‑cause analysis

log_format upstream_timing '$remote_addr $host "$request" $status '
    'request_time=$request_time '
    'upstream_addr=$upstream_addr '
    'upstream_status=$upstream_status '
    'upstream_connect_time=$upstream_connect_time '
    'upstream_header_time=$upstream_header_time '
    'upstream_response_time=$upstream_response_time';

This format captures total request time and per‑upstream metrics, enabling you to tell whether latency originates in Nginx or the backend.

2.4 Full site example (upstream, HTTPS server, and proxy)

upstream <site>_backend {
    server <backendIP1>:<port> max_fails=3 fail_timeout=10s;
    server <backendIP2>:<port> max_fails=3 fail_timeout=10s;
    keepalive 32;
}

server {
    listen 443 ssl;
    server_name <domain>;
    access_log /var/log/nginx/<site>-access.log upstream_timing;
    error_log  /var/log/nginx/<site>-error.log warn;
    location / {
        proxy_pass http://<site>_backend;
        proxy_http_version 1.1;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_connect_timeout 3s;
        proxy_read_timeout 30s;
    }
}

Remember to add the appropriate ssl_certificate directives for your environment.

3. Validate Traffic Distribution

3.1 Basic health check from the entry point

curl --fail --silent --show-error --location \
    --connect-timeout 3 --max-time 10 \
    https://<domain><healthPath>

The --fail flag makes HTTP 4xx/5xx return a non‑zero exit code, suitable for automation.

3.2 Record backend identifiers over multiple requests

for i in $(seq 1 20); do
  curl --silent --show-error --fail \
       -H 'Connection: close' \
       -D - -o /dev/null "https://<domain><healthPath>" \
       | grep -i '^X-Backend-Id:'
 done

Backends should emit a lightweight header (e.g., X-Backend-Id) in the health response so you can confirm that requests are hitting multiple instances.

3.3 Sticky sessions when required

upstream <site>_sticky_backend {
    ip_hash;
    server <backendIP1>:<port>;
    server <backendIP2>:<port>;
}

Use ip_hash only after evaluating the impact of NAT or proxy chains, which can concentrate traffic on a single node.

3.4 Least‑connections for heterogeneous request times

upstream <site>_least_conn_backend {
    least_conn;
    server <backendIP1>:<port>;
    server <backendIP2>:<port>;
}

It prefers the server with the fewest active connections, but you must verify its benefit with real metrics.

3.5 Consistent hashing for stable key mapping

upstream <site>_hash_backend {
    hash $request_uri consistent;
    server <backendIP1>:<port>;
    server <backendIP2>:<port>;
    server <backendIP3>:<port>;
}

Adding or removing a node re‑maps only a small fraction of keys, useful for cache sharding. Design the hash key around business semantics, not raw query strings.

4. Failure Handling

4.1 Passive failure thresholds and backup nodes

upstream <site>_backend {
    server <backendIP1>:<port> max_fails=3 fail_timeout=10s;
    server <backendIP2>:<port> max_fails=3 fail_timeout=10s;
    server <backupIP>:<port> backup;
}

If a primary node reaches max_fails within fail_timeout, Nginx temporarily skips it. The backup server only receives traffic when all primaries are down.

4.2 Control which upstream responses trigger a retry

location / {
    proxy_pass http://<site>_backend;
    proxy_next_upstream error timeout http_502 http_503 http_504;
    proxy_next_upstream_tries 2;
    proxy_next_upstream_timeout 5s;
}

Only idempotent or explicitly retry‑safe requests should use this; otherwise you risk duplicate writes.

4.3 Mark a node down for maintenance

upstream <site>_backend {
    server <backendIP1>:<port>;
    server <backendIP2>:<port> down;
    server <backendIP3>:<port>;
}

After marking down, verify that $upstream_addr no longer contains the node before proceeding with maintenance.

4.4 Extract error‑log evidence

sudo grep -E 'connect\(\) failed|upstream timed out|no live upstreams' \
    /var/log/nginx/<site>-error.log | tail -n 100

These messages indicate connection refusals, timeouts, or a completely unavailable upstream pool.

5. Observability

5.1 Enable stub_status on a restricted address

server {
    listen 127.0.0.1:8080;
    server_name localhost;
    location = /nginx_status {
        stub_status;
        allow 127.0.0.1;
        deny all;
    }
}

Expose it only on loopback and protect it with firewall rules; it provides connection and request counters but not per‑upstream metrics.

5.2 Verify stub_status is not reachable from the public network

curl --fail --silent http://127.0.0.1:8080/nginx_status
curl --fail --silent --max-time 3 http://<entryIP>:8080/nginx_status && exit 1 || true

The second command must fail; otherwise the status page is exposed.

5.3 Find the slowest backend responses from logs

sudo awk '
  match($0, /upstream_addr=([^ ]+)/, a) &&
  match($0, /upstream_response_time=([^ ]+)/, t) {
    print a[1], t[1]
  }
' /var/log/nginx/<site>-access.log |
  sort -k2,2nr | head -n 50

Combine the top‑slow entries with backend, path, status code, and deployment version before concluding a node is faulty.

6. Gray Release, Scaling, and Rollback

6.1 Pre‑check a new backend and perform a single‑node gray rollout

curl --fail --silent --max-time 3 http://<newIP>:<port><healthPath>
sudo nginx -t
sudo systemctl reload nginx
sudo tail -n 100 /var/log/nginx/<site>-access.log

Validate the new node can be reached, the config parses, and the access log shows the node receiving traffic.

6.2 Dynamic DNS resolution with resolver

resolver <dnsServerIP> valid=30s;
resolver_timeout 2s;
set $backend_url http://<backendDomain>:<port>;
location / { proxy_pass $backend_url; proxy_set_header Host $host; }

When proxy_pass uses a variable, Nginx requires a resolver. Verify DNS TTL, IPv4/IPv6 compatibility, and fallback behavior before using it in production.

6.3 Roll back to a previous configuration backup

#!/usr/bin/env bash
set -euo pipefail
CONF_FILE="<configPath>"
BACKUP_FILE="/var/backups/nginx/<configFile>.<timestamp>"
sudo test -r "${BACKUP_FILE}"
sudo cp -a "${BACKUP_FILE}" "${CONF_FILE}"
sudo nginx -t
sudo systemctl reload nginx
curl --fail --silent --max-time 10 "https://<domain><healthPath>"

After copying the backup, run a syntax check, reload, and perform an end‑to‑end health request.

6.4 Ensure no stale new‑backend IP remains after rollback

sudo nginx -T 2>&1 | grep -F '<newIP>' && exit 1 || true
sudo grep -F '<newIP>' /var/log/nginx/<site>-access.log | tail -n 20

If the first command still finds the IP, an include file was not rolled back; locate and correct it before reloading again.

7. Common Misconceptions

Restarting Nginx on every 502 is wrong; investigate error logs, upstream fields, and backend reachability first.

Treating max_fails as an active health check leads to delayed detection on low‑traffic services.

Enabling proxy_next_upstream for all locations can cause duplicate writes on non‑idempotent requests.

A successful nginx -t does not guarantee a successful deployment; verify traffic distribution and latency.

Exposing stub_status publicly leaks operational data; keep it behind strict access controls.

8. Acceptance Criteria

After an upstream change you must be able to answer:

Which configuration file is currently active?

Which backends are reachable and what algorithm is used?

How are connection failures, timeouts, and 5xx responses handled?

What is the per‑backend request distribution, latency, and status?

How is a node drained for maintenance?

If the new config fails, how do you revert to a specific backup?

Combining configuration files, automated tests, logs, and metrics provides a complete operational view of the upstream load‑balancing component.