From Single Server to Billion-User Scale: A Java Backend Architecture Evolution Guide

21st‑century technical guide by an Alibaba P9 architect describing how a Java website evolves from a single machine to supporting over a hundred million concurrent users.

Stage 1: Single‑Machine Site

Initially all components (Tomcat/Jetty, JSP/Servlet, Maven+Spring+Hibernate/MyBatis, MySQL/SQLServer/Oracle) run on one server, forming a small system.

Stage 2: Separate Application Server and Database

When traffic grows, the application server and database are split onto different machines to improve load capacity and fault tolerance.

Stage 3: Application Server Cluster

Multiple application servers are added; requests are distributed using load‑balancing software such as keepalived combined with ipvsadm.

Four key problems arise: request forwarding, forwarding algorithm, response handling, and session consistency.

Load‑balancing solutions (5 types)

HTTP redirect – simple but low performance.

DNS load balancing – no server maintenance but slow failure detection.

Reverse proxy (Apache, Nginx) – easy deployment, possible bottleneck.

IP‑layer load balancing – better performance, bandwidth bottleneck.

Data‑link layer load balancing – avoids returning through the balancer.

Scheduling algorithms (10 types)

Round‑robin (rr) – simple, ignores server capacity.

Weighted round‑robin (wrr) – considers server weight.

Source‑IP hash (sh) – maps same client IP to same server.

Destination‑IP hash (dh) – similar to sh but uses destination IP.

Least connections (lc) – prefers server with fewest connections.

Weighted least connections (wlc) – combines weight with lc.

Shortest expected delay (sed) – like wlc without idle connections.

Never‑queue (nq) – sends to a server with zero connections.

Locality‑based least connections (lblc) – prefers server recently used by the client IP.

Locality‑based least connections with replication (lblcr) – selects from a server group, falls back if overloaded.

Stage 4: Database Read‑Write Splitting

To avoid data inconsistency when scaling databases, master‑slave replication or middleware (e.g., Mycat) is used to separate reads from writes.

Stage 5: Search Engine for Read‑Heavy Queries

Search engines (inverted index) accelerate fuzzy searches (e.g., product title lookup) but add maintenance overhead and require a dedicated cluster.

Stage 6: Caching Layer

Application‑level caches (Guava, Memcached) and database caches (Redis) reduce database load; page caches (HTML5 localStorage, cookies) improve response speed. Consistent‑hashing is recommended for cache node selection.

Stage 7: Database Sharding

7.1 Vertical Sharding

Separate business domains (transactions, products, users) into different databases, improving performance but requiring handling of cross‑database joins and transactions.

7.2 Horizontal Sharding

Split a single table across multiple databases to overcome size or write‑throughput limits; requires SQL routing, primary‑key strategies (UUID), and pagination handling, often via middleware like Mycat.

Stage 8: Application Splitting

8.1 Split Applications

Divide a monolithic app into sub‑systems (e.g., user‑product, user‑transaction) to avoid code bloat.

8.2 Service‑Oriented Architecture (SOA)

Extract common functionalities into services, enabling reuse, clearer separation, and focusing front‑ends on browser interaction.

Stage 9: Message Middleware

Introduce language‑agnostic middleware (e.g., Dubbo with Zookeeper) to enable reliable inter‑service communication, load balancing, and monitoring across heterogeneous platforms.

Conclusion

The presented evolution is illustrative; real‑world architectures must be tailored to specific business needs and constraints, requiring continuous analysis and adaptation.