How Encyclopedia Systems Survive Data Center Fires: Architecture Deep Dive

The article explores how encyclopedia systems like Wikipedia handle billions of daily visits and survive catastrophic data center failures, using a hypothetical "Wepedia" design to illustrate the core reasoning.

Step 1: Calculate Network Latency

Why are multiple data centers mandatory? The answer lies in physics. Earth's circumference is 40,000 km; light in fiber travels at ~300,000 km/s. A round-trip across half the globe takes (40,000 km / 2) / 300,000 km/s * 2 ≈ 133 ms. Adding server processing pushes user wait time over 1 second — beyond the threshold where users become impatient. Therefore, multi-data-center deployment is a necessity, not a luxury. GeoDNS routes each user to the nearest data center, slashing network latency from ~150 ms to under 10 ms.

Step 2: Deconstruct the Five-Hop Request Chain

A user request for an encyclopedia entry travels through a defined sequence:

Browser queries GeoDNS → receives nearest CDN node IP.

CDN serves cached entry if available (request ends).

On cache miss, request hits LVS (Layer 4 load balancer).

LVS forwards to Nginx reverse proxy cluster.

Nginx checks its own cache; if still missing, passes via a second LVS to Apache/PHP application servers.

PHP queries Redis cache.

Finally, if all caches miss, MySQL database is queried.

Key insight: over 80% of requests are intercepted and returned by CDN and Nginx, never reaching the application servers. This explains how the architecture handles 1 billion daily visits with relatively few servers.

Step 3: Three-Layer Performance Optimization Pyramid

Frontend Layer (blocks 80%+ of traffic)

CDN caches hot entry pages.

Nginx reverse proxy cluster spans dozens of servers.

GeoDNS provides proximity-based routing.

Service Layer (accelerates code execution)

PHP uses APC opcode caching to speed up script execution.

Scientific formulas are rendered via TeX into images, replacing inefficient string-search algorithms.

Top-tier hardware is provisioned for application servers.

Storage Layer (multi-level caching to protect the database)

Hottest data cached directly in application server local memory.

General hot data stored in Redis cluster.

Persistent storage uses MySQL with RAID5 disk arrays and master-slave replication.

The three cache tiers progressively filter requests, minimizing database load.

Step 4: Master-Slave Multi-Data-Center Data Synchronization

The toughest challenge in multi-data-center setups is data consistency. Wepedia adopts a simple but effective single-master, multi-slave model:

All write requests (POST) are forwarded to the master data center.

After the master database updates, the Canal component captures changes and synchronizes them to MySQL instances in all slave data centers.

Read requests (GET) are served locally from any data center.

This design is justified by the workload: the GET-to-POST ratio exceeds 1000:1, so write operations are extremely rare and replication latency is imperceptible to nearly all users.

Disaster recovery is built in: if the master data center is destroyed (e.g., by fire), the remaining data centers run a leader election algorithm (similar to ZooKeeper) to automatically promote a new master. User traffic is seamlessly rerouted, keeping the system continuously available.

Reusable Four-Step Template

The article distills the above reasoning into a repeatable template for any read-heavy, globally distributed high-concurrency application:

Calculate latency — use physical formulas to quantify cross-region communication cost and justify multi-data-center deployment.

Deconstruct the request chain — map the full path from user to database, placing cache interception points at every layer.

Layered optimization — frontend blocks 80% of requests, service layer accelerates code, storage layer uses multi-level caching to minimize database pressure.

Solve consistency — employ single-master multi-slave with Canal synchronization to achieve multi-data-center consistency at minimal cost, while gaining failover capability.

One-Sentence Summary

The essence of high concurrency is not stacking more machines, but eliminating requests before they reach the core system through layers of caching.