Mastering Nginx WebSocket Reverse Proxy: From Basic Setup to Production‑Ready Architecture

1. WebSocket Core Principles

WebSocket uses the HTTP/1.1 Upgrade mechanism to establish a full‑duplex TCP channel. After a successful handshake, the connection no longer follows the request‑response model, making it highly sensitive to Nginx timeout, file‑descriptor, kernel, and load‑balancer settings.

2. Basic Reverse‑Proxy Configuration

http {
    upstream websocket_backend {
        server 192.168.1.100:8080;
        server 192.168.1.101:8080;
        keepalive 10;
    }
    server {
        listen 80;
        server_name ws.example.com;
        location /ws/ {
            proxy_pass http://websocket_backend;
            proxy_http_version 1.1;
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection "upgrade";
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;
            proxy_read_timeout 3600s;
            proxy_send_timeout 3600s;
            proxy_connect_timeout 30s;
            proxy_buffering off;
            proxy_cache off;
        }
    }
}

Key points: proxy_http_version 1.1, proper Upgrade / Connection headers, disabling buffering, and extending read/send timeouts to prevent premature disconnects.

3. Production‑Grade Upgrade Header Handling

⚠️ This is the most common pitfall in the original configuration.

map $http_upgrade $connection_upgrade {
    default upgrade;
    ''      close;
}

Then inside location:

proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $connection_upgrade;

Using the map ensures non‑WebSocket requests receive Connection: close, avoiding accidental upgrades.

4. Full Production Configuration

http {
    upstream websocket_cluster {
        ip_hash;
        server 10.0.1.10:8080 weight=3;
        server 10.0.1.11:8080 weight=2;
        server 10.0.1.12:8080 backup;
        # health checks require nginx_upstream_check_module
        check interval=3000 rise=2 fall=3 timeout=1000 type=http;
        check_http_send "GET /health HTTP/1.0

";
        check_http_expect_alive http_2xx http_3xx;
    }
    server {
        listen 443 ssl http2;
        server_name ws.example.com;
        ssl_certificate /etc/nginx/ssl/example.com.crt;
        ssl_certificate_key /etc/nginx/ssl/example.com.key;
        ssl_protocols TLSv1.2 TLSv1.3;
        location /chat/ {
            proxy_pass http://websocket_cluster;
            proxy_http_version 1.1;
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection $connection_upgrade;
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;
            proxy_read_timeout 86400s;
            proxy_send_timeout 86400s;
            proxy_buffering off;
            limit_conn ws_conn 1000;
        }
    }
}
limit_conn_zone $binary_remote_addr zone=ws_conn:10m;

5. HTTP/2 and WebSocket Compatibility

Browsers still use HTTP/1.1 for WebSocket. RFC 8441 defines HTTP/2 WebSocket, but support is limited and Nginx’s implementation is incomplete. Therefore, keep proxy_http_version 1.1 and explicitly configure the location for WebSocket.

6. Docker / Kubernetes Deployment

map $http_upgrade $connection_upgrade {
    default upgrade;
    ''      close;
}
server {
    listen 80;
    location /ws/ {
        proxy_pass http://websocket-service:8080;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection $connection_upgrade;
        proxy_read_timeout 7d;
        proxy_send_timeout 7d;
        proxy_buffering off;
    }
}

In cloud or K8s environments, idle‑timeout defaults (e.g., SLB 60 s, L4 LB 30‑120 s, Istio 1 h) can drop idle connections; application‑level heartbeats are mandatory.

7. Heartbeat Strategy

Send a ping message every 20‑30 seconds, e.g.: {"type":"ping"} Without regular heartbeats, load balancers, firewalls, or NAT devices will close idle WebSocket connections.

8. Load‑Balancing Choices

upstream websocket_backend {
    ip_hash;   # small scale
    # or least_conn for cloud/NAT heavy traffic
    # or application‑level registration (Redis) for large IM systems
}

WebSocket connections are not well‑suited to LB‑based session persistence.

9. Graceful Shutdown & Canary Release

terminationGracePeriodSeconds: 60

Receive SIGTERM.

Stop accepting new WebSocket connections.

Notify existing clients to reconnect.

Perform a smooth shutdown.

10. Nginx & Kernel Tuning for High Concurrency

worker_processes auto;
worker_rlimit_nofile 200000;
events {
    worker_connections 65535;
    use epoll;
}
net.core.somaxconn = 65535
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_fin_timeout = 15

11. Monitoring & Troubleshooting

wscat -c ws://domain/ws
ss -an | grep ESTAB | wc -l

12. Final Takeaways

WebSocket stability depends 70% on factors beyond Nginx: Application‑level heartbeat. Load‑balancer idle‑timeout settings. Graceful shutdown procedures. File‑descriptor and kernel parameter tuning.