Designing a Million‑Concurrent Backend: LVS, Nginx, and Spring Cloud Gateway

Million‑concurrent traffic is a core challenge for large‑scale architectures; this guide details a design that uses a multi‑layer load‑balancing approach.

Layered Load‑Balancing Design

Access Layer (LVS + Keepalived) : Use LVS (Layer‑4 virtual server) as the first line of defense, leveraging kernel‑mode forwarding for high performance and Keepalived for high availability to quickly distribute massive TCP/UDP connections.

After LVS, deploy Nginx for Layer‑7 load balancing, handling intelligent routing, static content caching, and SSL termination. Separating complex business logic from LVS improves its efficiency.

Gateway Layer : The gateway serves as the unified entry point for all microservices and must provide high performance and elasticity under million‑concurrent load.

Load‑Balancing Design

Under million‑concurrent load, a single‑layer load balancer (e.g., pure Nginx) cannot sustain peak traffic. Multi‑layer load balancing distributes network and application pressure.

upstream microservices {
    least_conn;
    server 192.168.1.10:8080 weight=3 max_fails=3 fail_timeout=10s;
    server 192.168.1.11:8080 weight=2;
    keepalive 32;
}
server {
    listen 80;
    location /api/ {
        proxy_pass http://microservices;
        proxy_set_header X-Real-IP $remote_addr;
        health_check interval=3 fails=2 uri=/actuator/health;
    }
}

Microservice Gateway Architecture

The gateway, as the unified entry for microservices, handles authentication, routing, and traffic shaping under million‑concurrent conditions. Spring Cloud Gateway (based on WebFlux and Netty) replaces Zuul, providing authentication, protocol conversion, route aggregation, and unified rate limiting.

Key points:

High‑concurrency architecture uses the Reactor model and asynchronous processing to avoid thread blocking.

Dynamic routing employs predicates (path/header matching) and filters (rate limiting, logging), integrating Nacos for service discovery and load balancing.

Core functions include JWT/OAuth authentication, gray release based on User‑Agent, and cache‑penetration protection.

Nginx offloads static resources while the gateway focuses on business routing.

Service Resilience

Circuit Breaking : Monitors service call metrics (error rate, latency) and automatically cuts off calls when thresholds are exceeded, preventing cascade failures. Common frameworks: Sentinel, Hystrix.

Rate Limiting and Degradation : Token bucket, leaky bucket, or counter algorithms control request rates to avoid resource exhaustion. Degradation reduces non‑critical functions to lightweight handling or static responses when resources are scarce, ensuring critical paths remain available.