Evolution of Server‑Side Architecture from Hundreds to Tens of Millions of Concurrent Users: A Taobao Case Study

Preface

The article uses Taobao as an illustrative example to show how a service‑side architecture evolves as traffic grows from a few hundred requests to tens of millions of concurrent users, and it lists the technical challenges encountered at each stage.

Basic Concepts

Distributed system: components deployed on different servers.

High availability: ability to keep serving when some nodes fail.

Cluster: multiple servers acting as a single service.

Load balancing: evenly distributing requests across nodes.

Forward and reverse proxy: mechanisms for internal‑external request routing.

Architecture Evolution

1. Single‑Machine Architecture

Initially Tomcat and the database run on the same server; sufficient for low traffic but becomes a bottleneck as users increase.

2. Separate Tomcat and Database

Deploy Tomcat and the database on separate machines, improving performance but making the database a new read/write bottleneck.

3. Introduce Local and Distributed Caches

Add in‑process caches (e.g., memcached) and an external Redis cache to offload most read traffic from the database, reducing DB pressure.

4. Reverse Proxy Load Balancing

Deploy multiple Tomcat instances behind Nginx or HAProxy, distributing requests and increasing concurrent capacity, while the database remains the next bottleneck.

5. Database Read‑Write Splitting

Use middleware such as Mycat to separate read and write databases, allowing multiple read replicas and improving read scalability.

6. Business‑Based Database Sharding

Allocate different business domains to separate databases, reducing contention but requiring cross‑database queries.

7. Table Partitioning (Horizontal Sharding)

Split large tables into smaller ones (hash‑by‑ID, time‑based tables) and use Mycat for routing, enabling horizontal scaling of the database layer.

8. Layer‑4 Load Balancing (LVS / F5)

Introduce LVS or hardware F5 devices to balance traffic among multiple Nginx instances, adding high‑availability via keepalived and virtual IPs.

9. DNS Round‑Robin Across Data Centers

Configure DNS to return multiple IPs, each pointing to a different data‑center, achieving geographic load balancing.

10. Adopt NoSQL and Search Engines

Integrate technologies such as HDFS, HBase, Redis, Elasticsearch, Kylin, and Druid to handle massive data, unstructured queries, and analytical workloads.

11. Split Monolith into Smaller Applications

Separate code bases by business domain, using Zookeeper for distributed configuration.

12. Extract Shared Functions into Microservices

Isolate common capabilities (user management, order, payment, authentication) into independent services using Dubbo or Spring Cloud for service governance.

13. Enterprise Service Bus (ESB)

Introduce an ESB to unify protocol conversion and reduce coupling between services, moving toward an SOA architecture.

14. Containerization

Package services as Docker images and orchestrate them with Kubernetes, enabling dynamic scaling and isolated runtime environments.

15. Cloud Platform Adoption

Deploy the whole system on public cloud (IaaS, PaaS, SaaS), leveraging elastic resources, managed services, and pay‑as‑you‑go to handle peak loads efficiently.

Design Summary and Principles

Design for N+1 redundancy to avoid single points of failure.

Include rollback and feature‑toggle mechanisms.

Plan monitoring and alerting from the start.

Prefer mature, commercially supported technologies.

Ensure horizontal scalability and resource isolation.

Adopt stateless service interfaces.

These guidelines help engineers evolve a system responsibly as traffic and business complexity grow.