How to Build a Billion‑Scale Open Platform: Architecture, Caching, and Resilience

Overall Architecture – Three‑Layer Decoupling and Traffic Partitioning

A mature open platform must sustain massive API traffic and concurrent requests. The solution is a three‑layer architecture that isolates responsibilities, enables independent scaling, and provides traffic governance.

Access Layer: API gateway, routing, authentication, and traffic‑control components (e.g., Nginx, OAuth2, token‑bucket rate limiting).

Capability Layer: Business‑logic orchestration and micro‑service coordination (e.g., Spring Cloud, Dubbo, Kafka). Service‑mesh features such as circuit‑breaker, retry and gray‑release are applied here.

Infrastructure Layer: Persistent storage, caching, messaging and monitoring (e.g., MySQL, Redis, RocketMQ, Prometheus, SkyWalking).

External callers → Access Layer → Capability Layer → Infrastructure Layer, each tier can be scaled horizontally without affecting the others.

Cache System Design & Hotspot Isolation

To keep latency low under billions of reads, a three‑level cache hierarchy is employed:

L1 – Local in‑process cache: In‑memory structures such as Caffeine provide nanosecond‑level reads limited to a single JVM instance.

L2 – Distributed cache: A Redis cluster shared by all service instances offers high concurrency, persistence and cross‑instance data sharing.

L3 – CDN edge cache: Static assets (videos, images) are cached at CDN edge nodes to offload the origin.

Typical request flow: client → L1 hit → return; on miss, L2 hit → return; otherwise the request falls back to the database. To protect the database during peak events (e.g., Double‑11), the design adds:

Bloom‑filter pre‑check to reject non‑existent keys.

Stale‑data fallback with asynchronous refresh to avoid cache‑stampede.

Write‑through or write‑behind strategies for consistency.

Asynchronous Architecture – Message Queues as Traffic Buffers

Synchronously chained services become a bottleneck under high load. Introducing a message queue decouples modules into an event‑driven pipeline, achieving "peak shaving":

Peak buffering: Incoming requests are written to a queue (e.g., Kafka) and processed asynchronously, smoothing traffic spikes.

Publish‑subscribe decoupling: Producers return immediately; consumers handle business logic later.

Failure handling: Automatic retries and dead‑letter queues capture unprocessed messages for manual inspection.

Database Elasticity – Sharding, Read‑Write Separation, and Hot‑Table Governance

Horizontal sharding expands capacity, while read‑write separation offloads query traffic from the primary database. A globally unique ID scheme guarantees ordered, conflict‑free primary keys across shards.

Cache pre‑aggregation: Frequently accessed hot data (e.g., leaderboard top‑N) is pre‑computed and stored in Redis.

Sharding & partitioning: Tables are split vertically by business domain or horizontally by time/key range to distribute load.

Batch asynchronous writes: High‑frequency updates are buffered in cache and flushed to the database in bulk, reducing write amplification.

Distributed Transaction Handling – Consistency vs. Performance

Micro‑service decomposition introduces cross‑service transaction challenges. Common patterns include:

TCC (Try‑Confirm‑Cancel): Explicit resource reservation with a compensating cancel step.

Saga: A sequence of local transactions with compensating actions for rollback.

AT (Automatic Transaction): Short‑lived DB‑native transactions using rollback logs.

Seata framework: Alibaba’s open‑source solution supporting AT, TCC, Saga and XA modes.

Distributed transactions should be used sparingly; idempotent designs, eventual consistency, or compensation logic are preferred whenever possible.

High‑Availability Architecture – Elastic Scaling

Container‑based auto‑scaling: Kubernetes Horizontal Pod Autoscaler (HPA) expands pods based on CPU/memory metrics, handling sudden traffic bursts.

Active‑active multi‑region deployment: Identical service instances run in separate data centers; traffic is routed to the nearest node, providing fault tolerance and lower latency.

Chaos engineering & fault injection: Deliberate termination of instances, network partition simulation, and database fault injection verify system resilience.

Implementation Roadmap – Phased Evolution to Billion‑Scale

Phase 1 – Foundation (0→1): Expose core APIs, launch a minimal developer portal, achieve small‑scale high availability (thousands of QPS).

Phase 2 – Performance Boost (1→10): Introduce multi‑level caching and asynchronous processing, split monolith into micro‑services, refine traffic governance, reach tens of thousands of QPS.

Phase 3 – Ecosystem Expansion (10→N): Build an ability marketplace, push throughput to millions of QPS, implement fine‑grained API versioning and advanced operational tooling.

Each stage emphasizes incremental refactoring, continuous monitoring, automated testing, and progressive capacity planning to ensure stability while scaling.