Mastering Nginx Reverse Proxy: From Basics to Advanced Load Balancing and High Availability

Essence of Reverse Proxy

A forward proxy sits between the client and the internet, while a reverse proxy sits in front of one or more backend servers. The client never sees the real servers; the proxy receives the request, parses the protocol, selects a backend via load‑balancing, forwards the request, processes the response, optionally caches it, logs the transaction and finally returns the response to the client.

Forward Proxy (client → proxy → target)
    • Bypass censorship, caching, anonymity
Reverse Proxy (client → proxy → backend)
    • Load balancing, security, cache acceleration

Basic Reverse Proxy Configuration

Minimal Proxy

server {
    listen 80;
    server_name example.com;
    location / {
        proxy_pass http://127.0.0.1:8080;
    }
}

Full Proxy Configuration

server {
    listen 80;
    server_name example.com;
    location / {
        proxy_pass http://backend;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header X-Forwarded-Host $host;
        proxy_set_header X-Forwarded-Port $server_port;
        proxy_http_version 1.1;
        proxy_set_header Connection "";
        proxy_connect_timeout 60s;
        proxy_send_timeout 60s;
        proxy_read_timeout 60s;
        proxy_buffering on;
        proxy_buffer_size 4k;
        proxy_buffers 8 4k;
        proxy_busy_buffers_size 8k;
        proxy_intercept_errors on;
        proxy_next_upstream error timeout http_502 http_503 http_504;
    }
}

Proxy Header Details

# Example request headers
Host: example.com               # original host
X-Real-IP: 203.0.113.10        # client IP
X-Forwarded-For: 203.0.113.10, 10.0.0.1, 192.168.1.1
X-Forwarded-Proto: https
# Test with:
curl -v http://example.com | grep -E "^> (Host|X-Real-IP|X-Forwarded)"

Log Configuration

log_format proxy_log '$remote_addr - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent" rt=$request_time uct="$upstream_connect_time" uht="$upstream_header_time" urt="$upstream_response_time" u="$upstream_addr"';
access_log /var/log/nginx/proxy.log proxy_log;

server {
    listen 80;
    server_name example.com;
    location / {
        proxy_pass http://backend;
        access_log off;               # only log errors
    }
    location /api/ {
        proxy_pass http://api_backend;
        access_log /var/log/nginx/api.log proxy_log;
    }
}

Load‑Balancing Strategies

Round Robin (default)

upstream backend {
    server 127.0.0.1:8080;
    server 127.0.0.1:8081;
    server 127.0.0.1:8082;
}
server {
    listen 80;
    server_name example.com;
    location / { proxy_pass http://backend; }
}

Weighted Round Robin

upstream backend {
    server 127.0.0.1:8080 weight=5;
    server 127.0.0.1:8081 weight=2;
    server 127.0.0.1:8082 weight=1;
}
# Distribution: 62.5% / 25% / 12.5%

IP Hash (session persistence)

upstream backend {
    ip_hash;
    server 127.0.0.1:8080;
    server 127.0.0.1:8081;
    server 127.0.0.1:8082;
}

Least Connections

upstream backend {
    least_conn;
    server 127.0.0.1:8080;
    server 127.0.0.1:8081;
    server 127.0.0.1:8082;
}

Generic Hash

upstream backend {
    hash $request_uri consistent;
    server 127.0.0.1:8080;
    server 127.0.0.1:8081;
}

Random

upstream backend {
    random two [method=round_robin];
    server 127.0.0.1:8080;
    server 127.0.0.1:8081;
    server 127.0.0.1:8082;
    server 127.0.0.1:8083;
}

Algorithm Comparison

round_robin – default, even distribution; best when backends have similar performance.

weighted – weight‑based distribution; useful for heterogeneous server capacities.

ip_hash – session persistence by client IP; suited for stateful services.

least_conn – directs traffic to the server with the fewest active connections; ideal for long‑lived connections.

hash – custom key (e.g., $request_uri, $cookie_jsessionid); helps with cache hit optimization.

random – random selection; works for stateless services.

Health‑Check Configuration

Passive Health Check

upstream backend {
    server 127.0.0.1:8080 max_fails=3 fail_timeout=30s;
    server 127.0.0.1:8081 max_fails=3 fail_timeout=30s;
    # max_fails: failures before marking down
    # fail_timeout: how long the server stays down
}

Active Health Check (third‑party module)

# Requires nginx‑upstream‑check‑module
upstream backend {
    zone backend 64k;
    check interval=3000 rise=2 fall=3 timeout=1000 type=http;
    check_http_send "GET /health HTTP/1.0

";
    check_http_expect_alive http_2xx http_3xx;
    server 127.0.0.1:8080;
    server 127.0.0.1:8081;
}

Health‑Check Endpoint

server {
    listen 8080;
    server_name _;
    location /health {
        access_log off;
        return 200 "OK
";
        add_header Content-Type text/plain;
    }
    location /ready {
        access_log off;
        set $ready 1;
        if ($database_connected = 0) { set $ready 0; }
        if ($ready = 0) { return 503; }
        return 200 "Ready
";
    }
    location / { # application code }
}

Session Persistence Mechanisms

Options

ip_hash – simple IP‑based affinity (limited in NAT environments).

Cookie hash – use a backend‑set cookie as the hash key, e.g. hash $cookie_PHPSESSID consistent;.

URI parameter hash – hash on a query parameter or another cookie, e.g. hash $cookie_jsessionid;.

Sticky module – sticky cookie srv_id expires=1h domain=.example.com path=/; for precise affinity.

Distributed Session Solutions

When multiple backends need shared session state, store sessions in an external store (Redis, Memcached) and let the application read/write via a cookie or token (JWT). Example for PHP:

session.save_handler = redis
session.save_path = "tcp://127.0.0.1:6379?database=0"
# Or at runtime
<?php
ini_set('session.save_handler', 'redis');
ini_set('session.save_path', 'tcp://127.0.0.1:6379?database=0');
?>

Java Spring Session example (application.yml):

# spring:
#   data:
#     redis:
#       host: localhost
#       port: 6379

Proxy Caching Configuration

Cache Workflow

Client → Nginx cache check → HIT returns cached response; MISS forwards to backend, stores the response, then returns it.

Cache Settings

proxy_cache_path /var/cache/nginx levels=1:2 keys_zone=api_cache:100m max_size=10g inactive=60m use_temp_path=off;
proxy_cache_key "$scheme$request_method$host$request_uri";
server {
    listen 80;
    server_name example.com;
    location / {
        proxy_pass http://backend;
        proxy_cache api_cache;
        proxy_cache_valid 200 60m;
        proxy_cache_valid 404 10m;
        proxy_cache_valid any 5m;
        proxy_cache_use_stale error timeout updating http_500 http_502 http_503 http_504;
        proxy_cache_background_update on;
        proxy_cache_lock on;
        proxy_cache_bypass $cookie_nocache $arg_nocache;
        proxy_no_cache $cookie_nocache $arg_nocache;
        add_header X-Cache-Status $upstream_cache_status;
    }
}
# $upstream_cache_status values: MISS, HIT, EXPIRED, STALE, UPDATING, REVALIDATED

Cache Purge (requires ngx_cache_purge)

server {
    listen 80;
    server_name example.com;
    location / { proxy_pass http://backend; proxy_cache api_cache; }
    location ~ /purge(/.*) {
        proxy_cache_purge api_cache "$scheme$request_method$host$1";
        allow 127.0.0.1;
        allow 10.0.0.0/8;
        deny all;
    }
}
# Example purge: curl -X PURGE http://example.com/api/data

Full Cache Example (API + static)

proxy_cache_path /var/cache/nginx/api levels=1:2 keys_zone=api_cache:100m max_size=10g inactive=7d use_temp_path=off;
proxy_cache_path /var/cache/nginx/static levels=1:2 keys_zone=static_cache:50m max_size=5g inactive=30d;
server {
    listen 80;
    server_name example.com;
    location /api/ {
        proxy_pass http://api_backend;
        proxy_cache api_cache;
        proxy_cache_valid 200 5m;
        proxy_cache_valid 404 1m;
        proxy_cache_use_stale error timeout updating;
        proxy_cache_lock on;
        add_header X-Cache-Status $upstream_cache_status;
        proxy_cache_methods GET HEAD POST;
        proxy_cache_bypass $cookie_nocache $arg_nocache $arg_cache;
        proxy_no_cache $cookie_nocache $arg_nocache $request_method;
    }
    location /static/ {
        proxy_pass http://static_backend;
        proxy_cache static_cache;
        proxy_cache_valid 200 30d;
        proxy_cache_valid 404 1m;
        expires 30d;
        add_header Cache-Control "public, no-transform";
    }
    location /api/private/ {
        proxy_pass http://api_backend;
        proxy_cache_bypass 1;
        proxy_no_cache 1;
    }
}

High‑Availability Solutions

Keepalived + Nginx (VRRP)

# Master node keepalived.conf
vrrp_script chk_nginx {
    script "/usr/local/bin/check_nginx.sh";
    interval 2;
    weight -20;
}
vrrp_instance VI_1 {
    state MASTER;
    interface eth0;
    virtual_router_id 51;
    priority 100;
    advert_int 1;
    authentication { auth_type PASS; auth_pass 1234; }
    virtual_ipaddress { 192.168.1.100; }
    track_script { chk_nginx; }
}
# Backup node is identical but state BACKUP and lower priority.

# /usr/local/bin/check_nginx.sh
#!/bin/bash
nginx_status=$(curl -s -o /dev/null -w "%{http_code}" http://localhost/health)
if [ $nginx_status -ne 200 ]; then
    systemctl restart nginx
    sleep 2
    nginx_status=$(curl -s -o /dev/null -w "%{http_code}" http://localhost/health)
    if [ $nginx_status -ne 200 ]; then exit 1; fi
fi
exit 0

DNS Round‑Robin HA

Configure multiple A records for the same hostname (e.g., example.com A 192.168.1.101, ... A 192.168.1.102, ... A 192.168.1.103). Simple but suffers from DNS caching and slower failover.

Consul + Nginx Dynamic Upstream

# consul-template upstream.ctmpl
upstream backend {
    {{ range service "api" }}
    server {{ .Address }}:{{ .Port }};
    {{ end }}
}
server {
    listen 80;
    server_name example.com;
    location / { proxy_pass http://backend; }
}

Real‑World Scenarios

API Gateway

upstream user_service { server 127.0.0.1:8001; server 127.0.0.1:8002; }
upstream order_service { server 127.0.0.1:8003; server 127.0.0.1:8004; }
upstream payment_service { server 127.0.0.1:8005; }
proxy_cache_path /var/cache/nginx levels=1:2 keys_zone=api_cache:100m max_size=10g inactive=5m;
server {
    listen 80;
    server_name api.example.com;
    location /api/users { proxy_pass http://user_service; limit_req zone=api burst=50 nodelay; }
    location /api/orders { proxy_pass http://order_service; limit_req zone=api burst=30 nodelay; proxy_cache api_cache; proxy_cache_valid 200 1m; proxy_cache_bypass $cookie_nocache; }
    location /api/payment { proxy_pass http://payment_service; limit_req zone=payment burst=5 nodelay; proxy_cache_bypass 1; proxy_no_cache 1; }
    location / { return 404; }
}

Single‑Page Application (SPA)

server {
    listen 80;
    server_name app.example.com;
    root /var/www/spa/dist;
    index index.html;
    location ~* \.(js|css|png|jpg|jpeg|gif|ico|svg|woff|woff2)$ {
        expires 1y;
        add_header Cache-Control "public, immutable";
        access_log off;
    }
    location = /index.html { expires -1; add_header Cache-Control "no-store, no-cache, must-revalidate"; }
    location / { try_files $uri $uri/ /index.html; }
    location /api/ { proxy_pass http://api_backend; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; }
}

WebSocket Proxy

map $http_upgrade $connection_upgrade { default upgrade; '' close; }
upstream websocket_backend { server 127.0.0.1:8080; server 127.0.0.1:8081; }
server {
    listen 80;
    server_name ws.example.com;
    location /ws {
        proxy_pass http://websocket_backend;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection $connection_upgrade;
        proxy_connect_timeout 7d;
        proxy_send_timeout 7d;
        proxy_read_timeout 7d;
        proxy_buffering off;
    }
}

gRPC Proxy

upstream grpc_backend { server 127.0.0.1:50051; server 127.0.0.1:50052; }
server {
    listen 443 ssl http2;
    server_name grpc.example.com;
    ssl_certificate /etc/nginx/ssl/grpc.crt;
    ssl_certificate_key /etc/nginx/ssl/grpc.key;
    ssl_protocols TLSv1.2 TLSv1.3;
    location / {
        grpc_pass grpc://grpc_backend;
        error_page 502 = /error502grpc;
    }
    location = /error502grpc {
        internal;
        default_type application/grpc;
        add_header grpc-status 14;
        add_header content-type application/grpc;
        add_header grpc-message "unavailable";
        return 204;
    }
}

Performance Optimisation

Connection Pool

upstream backend {
    server 127.0.0.1:8080;
    server 127.0.0.1:8081;
    keepalive 32;               # number of idle keep‑alive connections
    keepalive_requests 1000;
    keepalive_timeout 60s;
}
server {
    listen 80;
    location / { proxy_pass http://backend; proxy_http_version 1.1; proxy_set_header Connection ""; }
}

Buffer Optimisation

proxy_buffering on;
proxy_buffer_size 4k;
proxy_buffers 8 4k;
proxy_busy_buffers_size 16k;
proxy_max_temp_file_size 1024m;
proxy_temp_file_write_size 8k;
fastcgi_buffering on;
fastcgi_buffer_size 4k;
fastcgi_buffers 8 4k;
fastcgi_busy_buffers_size 8k;

Zero‑Copy Transfer

sendfile on;
tcp_nopush on;
tcp_nodelay on;
aio on;
directio 4m;
output_buffers 1 128k;

Gzip Compression

gzip on;
gzip_vary on;
gzip_proxied any;
gzip_comp_level 6;
gzip_min_length 1024;
gzip_types text/plain text/css text/xml application/json application/javascript application/xml application/xml+rss image/svg+xml;
gzip_disable "msie6";

Monitoring and Troubleshooting

Metrics Collection (requires nginx‑module‑vts)

#!/bin/bash
echo "=== Upstream Health ==="
curl -s http://localhost/status

echo "
=== Response Time Test ==="
time curl -s -o /dev/null http://localhost/health

echo "
=== Upstream Availability ==="
for port in 8080 8081 8082; do
  if timeout 1 bash -c "echo > /dev/tcp/127.0.0.1/$port" 2>/dev/null; then
    echo "✓ 127.0.0.1:$port available"
  else
    echo "✗ 127.0.0.1:$port unavailable"
  fi
done

echo "
=== Recent Proxy Errors ==="
tail -20 /var/log/nginx/error.log | grep -E "upstream|proxy" || echo "No errors"

Common Error Troubleshooting

# 502 Bad Gateway – verify backend process is running and reachable.
ss -tlnp | grep :8080
curl -v http://127.0.0.1:8080
tail -50 /var/log/nginx/error.log | grep upstream

# 504 Gateway Timeout – check upstream response times (uht/urt fields).

# 503 Service Unavailable – examine rate‑limit settings and backend load.

# Uneven load – analyse $upstream_addr in access logs:
awk '{print $NF}' /var/log/nginx/access.log | sort | uniq -c | sort -rn

Debug Configuration

server {
    listen 80;
    server_name debug.example.com;
    location / { rewrite ^(.*)$ /debug$1 break; }
    location /debug/ {
        proxy_pass http://backend;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        add_header X-Upstream-Addr $upstream_addr;
        add_header X-Upstream-Status $upstream_status;
        add_header X-Upstream-Response-Time $upstream_response_time;
    }
}

Cheat Sheet

Core Configuration

# Basic reverse proxy
location / {
    proxy_pass http://backend;
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_http_version 1.1;
    proxy_set_header Connection "";
}

# Load balancing (choose one)
upstream backend {
    server 127.0.0.1:8080;
    server 127.0.0.1:8081 weight=2;
    ip_hash;
    least_conn;
}

# Passive health check
upstream backend {
    server 127.0.0.1:8080 max_fails=3 fail_timeout=30s;
    server 127.0.0.1:8081 max_fails=3 fail_timeout=30s;
}

# Caching
proxy_cache_path /var/cache/nginx levels=1:2 keys_zone=api_cache:100m;
location / {
    proxy_cache api_cache;
    proxy_cache_valid 200 60m;
}

# WebSocket
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";

Scenario Snippets

API Gateway – multiple upstreams, per‑location rate limiting, selective caching.

SPA – try_files $uri $uri/ /index.html, long‑term static cache.

WebSocket – Upgrade header, 7‑day timeouts.

gRPC – listen on 443 with http2, grpc_pass.

High Availability – Keepalived VRRP, DNS round‑robin, Consul dynamic upstream.

Monitoring Commands

# View upstream status (vts module)
curl http://localhost/status
# Tail recent errors
tail -100 /var/log/nginx/error.log | grep upstream
# Test health endpoint
curl -v http://127.0.0.1:8080/health
# Show active connections
ss -ant | grep :8080
# Analyse request distribution
awk '{print $NF}' /var/log/nginx/access.log | sort | uniq -c | sort -rn

Performance Tips

1. Connection pool: keepalive 32‑64
2. Buffering: proxy_buffering on; tune sizes
3. Zero‑copy: sendfile on + tcp_nopush
4. Compression: gzip on with appropriate types
5. HTTP/2 for TLS endpoints
6. Cache frequently accessed content to reduce backend load