Why Single‑Server Apps Fail: Master Load Balancing with Nginx and LVS

My technical blog: www.susan.net.cn

Hello, I am Su San. Today I’ll briefly discuss load balancing, revising a previously published article with new insights.

Many interviewers ask for a detailed description of the request flow from entering a keyword on Taobao to the final page. Answering this requires knowledge of HTTP, TCP, gateways, LVS, and related protocols. Understanding each component illuminates how traffic moves and helps troubleshoot issues.

Initially, Li DaNiu’s startup deployed a single Tomcat server, which handled all traffic because the load was low. As the business grew, the single server became a bottleneck and a single point of failure.

To address this, multiple Tomcat instances (e.g., three) were added, and client requests were distributed among them. However, letting clients choose a server directly is problematic because they cannot detect failed instances.

Therefore, a dedicated load‑balancer (LB) layer was introduced, typically using Nginx, to receive all client requests and forward them to healthy backend servers based on a load‑balancing algorithm.

To improve security, a gateway layer was placed in front of the LB to perform authentication, risk control, protocol conversion, and traffic shaping before traffic reaches the servers.

Static resources (JS, CSS, images) were later offloaded from Tomcat to Nginx, leveraging Nginx’s proxy cache to reduce disk I/O and improve performance.

This separation of dynamic and static requests—dynamic traffic goes through the gateway to Tomcat, while static traffic is served directly by Nginx—allows each component to focus on its strengths.

For internal services that do not require gateway authentication, separate Tomcat instances were deployed and accessed directly via Nginx, bypassing the gateway.

To avoid a single point of failure for Nginx itself, two Nginx instances were deployed in active‑backup mode using keepalived, which monitors heartbeat packets and promotes the standby node when the primary fails.

How does a four‑layer load balancer work?

LVS, a layer‑4 load balancer, forwards packets without establishing separate TCP connections to upstream servers, resulting in higher throughput and lower resource consumption compared to layer‑7 Nginx.

Combining both, the architecture places LVS in front of Nginx: LVS handles high‑volume packet forwarding, while Nginx manages application‑level processing.

For further scalability, multiple LVS instances can be used with DNS‑based load balancing, distributing client queries across LVS nodes.

If LVS can be replaced by multiple Nginx instances with layer‑4 support, is LVS still necessary?

While Nginx 1.9+ supports layer‑4 balancing, LVS operates in kernel space, avoiding user‑space context switches and offering superior performance for massive concurrency, making it a better fit for ultra‑high‑traffic scenarios.

Finally, static assets should be served via a CDN to reduce latency and offload traffic from the origin servers.

Summary

Architecture must align with business needs; small‑to‑medium traffic can rely on Nginx alone, but rapid growth warrants adding LVS + Nginx, and extreme traffic (tens of Gbps, millions of concurrent connections) may require custom layer‑4 solutions. Layered design isolates responsibilities, simplifies scaling, and mirrors the TCP/IP model where each layer handles its own concerns.