How to Scale a Web Service: From Single Tomcat to LVS + Nginx Architecture

Everyone has heard the classic interview question: "Describe the entire process from entering a keyword on Taobao to the final web page display, in as much detail as possible." This question touches on HTTP, TCP, gateways, LVS, and many other protocols.

Initially, Li DaNiu deployed a single Tomcat server, which handled all client requests directly. This worked while traffic was low, but as the business grew, a single machine became a bottleneck and a single point of failure.

To solve this, multiple Tomcat instances (e.g., three) were added, and client requests were distributed randomly. However, letting clients choose servers is impractical because they would need to know the server list and cannot detect failures.

The solution is to introduce a Load Balancer (LB) layer, typically Nginx, which receives all client traffic and forwards it to the appropriate backend server.

To improve security, a gateway layer was added before the servers for authentication, risk control, protocol conversion, and traffic shaping.

Static resources (JS, CSS) were later offloaded to Nginx because Tomcat is inefficient at serving static files, and Nginx’s proxy cache greatly improves performance.

Dynamic‑static separation was implemented: dynamic requests go through the gateway to Tomcat, while static requests are served directly by Nginx.

Because Nginx operates at layer 7, it must establish TCP connections both with clients and upstream servers, consuming memory and limiting load capacity. To overcome this, a layer‑4 load balancer (LVS) was introduced, which forwards packets without creating connections, offering higher performance and lower resource usage.

How does a layer‑4 load balancer work?

LVS forwards the client’s SYN packet to the selected backend after rewriting the destination IP, letting the client and server complete the TCP handshake directly.

Combining LVS with Nginx provides both high‑performance packet forwarding and advanced application‑level features. LVS is deployed in active‑standby mode, and Nginx is also duplicated with keepalived for failover.

To avoid a single point of failure for LVS, multiple LVS instances are used with DNS load balancing, distributing client queries across them.

For very large traffic, static assets should be placed behind a CDN, which serves content from the nearest node to the user.

Summary

Architecture must be tailored to business needs; small‑to‑medium traffic can rely on Nginx alone, while rapid growth may require adding LVS, and massive traffic (tens of Gbps) may need custom layer‑4 solutions. Layered design ensures each component focuses on its responsibilities, simplifying expansion and maintenance.