From a Simple MVP Monolith to a Complex Distributed Architecture: Taobao Case Study

A simple MVP‑style monolithic architecture can gradually evolve into a complex distributed system. Using a simulated Taobao example, the article demonstrates how a system scales from a few hundred concurrent users to millions, highlighting the technical challenges and solutions at each stage.

Basic Concepts

Distributed system: modules deployed on different servers.

High availability: failed nodes are seamlessly taken over by others.

Cluster: multiple servers providing a unified service, with automatic failover.

Load balancing: evenly distributing requests across nodes.

Forward and reverse proxy: forwarding internal requests outward and directing external requests inward.

Architecture Evolution

1. Single‑machine architecture

Initially, Tomcat and the database run on the same server; DNS resolves www.taobao.com to an IP that points to this Tomcat.

2. Separate Tomcat and database

Tomcat and the database are deployed on separate machines, improving resource isolation but making the database a bottleneck as traffic grows.

3. Introduce local and distributed cache

Local cache (e.g., memcached) and distributed cache (Redis) store hot items, dramatically reducing database load.

4. Add reverse proxy for load balancing

Deploy multiple Tomcat instances behind Nginx or HAProxy, distributing requests and increasing concurrent capacity.

5. Database read/write separation

Use a middleware such as MyCAT to split reads and writes, adding read replicas and improving read scalability.

6. Business‑based database sharding

Separate data per business into different databases, reducing contention and enabling horizontal scaling.

7. Split large tables into small tables

Hash‑based or time‑based partitioning creates many small tables; MyCAT and MPP databases (e.g., TiDB, Greenplum) handle the logical distributed database.

8. LVS/F5 for multi‑Nginx load balancing

Layer‑4 load balancers (LVS software or F5 hardware) distribute traffic among multiple Nginx instances, providing higher throughput and high availability.

9. DNS round‑robin for inter‑datacenter balancing

Configure DNS to return multiple IPs, each pointing to a different data‑center, achieving geographic load distribution.

10. Introduce NoSQL and search engines

Adopt HDFS, HBase, Redis, Elasticsearch, Kylin, Druid, etc., to handle massive data, full‑text search, and analytical workloads.

11. Split monolith into small applications

Divide code by business domain, allowing independent development and deployment.

12. Extract shared functions into microservices

Common capabilities (user management, order, payment, authentication) become independent services accessed via HTTP, TCP, or RPC, managed with frameworks like Dubbo or Spring Cloud.

13. Use an Enterprise Service Bus (ESB)

ESB unifies protocol conversion and service interaction, reducing coupling and enabling SOA‑style architecture.

14. Containerization and cloud platform

Docker packages services; Kubernetes orchestrates containers; moving to public cloud (IaaS/PaaS/SaaS) provides elastic resources, reducing operational cost.

Architecture Design Summary

The evolution path is not mandatory; real‑world constraints dictate which steps to take. For a one‑off system, design to meet performance targets with room for future expansion; for continuously growing platforms, design for the next growth stage and iterate.

Key differences between service‑side architecture and big‑data architecture are clarified: big‑data focuses on data ingestion, storage, processing, and analytics, while service architecture concerns application organization built atop those data capabilities.

Design Principles (12 items)

N+1 design – no single point of failure.

Rollback capability – ensure forward compatibility.

Feature toggle – allow quick disabling of problematic functions.

Monitoring – embed observability from the start.

Active‑active data centers for high availability.

Prefer mature, commercially supported technologies.

Resource isolation – prevent one business from monopolizing resources.

Horizontal scalability – design for scale‑out.

Buy non‑core solutions when appropriate.

Use enterprise‑grade hardware.

Rapid iteration – develop small features quickly for feedback.

Stateless services – avoid reliance on previous request state.

Author : Shi Hua, experienced in big‑data technologies and architecture, with years of practice in high‑concurrency and distributed systems.