Final Episode: Building a Million‑Concurrent Product System

Module Review

The series concludes after fourteen articles, summarizing each module’s core deliverables from demand background to technical design, data migration, caching architecture, launch assurance, and traffic isolation for a complete C‑end product system.

Module 1: Demand and Design (Articles 2‑3)

The requirement document explains that a major promotion incident exposed the PHP system’s inability to scale, prompting a redesign. The technical solution proposes a five‑service collaborative architecture with strict rollout milestones, establishing a global view that underpins later decisions.

Module 2: Migration Engineering (Articles 4‑7)

Four articles detail the full migration from PHP to Java. A traffic‑hijack service enables seamless cut‑over for dozens of callers; an issue with Spring MVC’s strict HTTP validation led to reverting to a Servlet implementation. The product‑admin service implements an EAV model with five tables and dynamic attributes. The data‑migration service moves tens of billions of rows from old to new tables, using full‑plus‑incremental loads to avoid loss or duplication, while quality checks examine record counts, missing rows, extra rows, and field values.

Module 3: Product Read Service (Articles 8‑12)

This is the most technically intensive part, constructing a high‑concurrency read service across five articles. The read API uses an Option pattern to return only requested data. A Loader mechanism provides separate loaders per data dimension, with both hand‑written and framework‑based implementations. Off‑heap cache (OHC) stores data outside the JVM heap, avoiding GC impact and serving as a temporary acceleration layer during promotions. A three‑layer Redis cache assigns responsibilities: empty‑value cache prevents penetration, active refresh mitigates cache breakdown, and weighted TTL disperses hot‑key load. Cache updates are driven by a message queue, and a custom hit‑rate statistic tool assists tuning.

Module 4: Launch Assurance (Articles 13‑14)

Real‑traffic end‑to‑end interface validation uncovers issues that database‑level checks miss, such as EAV aggregation logic errors, field‑mapping mismatches, and serialization precision problems. For a 60× traffic surge in the collection domain, twenty dedicated machines provide physical isolation, with Nginx routing based on caller identifiers.

High‑Concurrency Topics Covered

Off‑heap cache (OHC): serialization, memory management, and GC isolation when data exceeds JVM heap.

Three‑layer cache hierarchy: coordination between OHC, Redis, and the database.

Cache‑penetration protection: empty‑value caching and Bloom filter applicability.

Cache‑breakdown protection: active refresh for hot data spikes.

Hot‑key handling: distribution strategies for high‑frequency Redis node access.

Cache pre‑warming: preparing caches before large promotions to avoid cold starts.

Cache update consistency: asynchronous refresh via message queues.

Hit‑rate statistics: hand‑written metrics for cache tuning.

Physical traffic isolation: separating the fastest‑growing callers onto dedicated machine pools.

High‑concurrency read‑API design: Option pattern reduces unnecessary computation and bandwidth.

Seamless traffic switching: proxy‑layer gray‑release enables zero‑code changes for dozens of callers.

Code Availability

The accompanying project provides complete, runnable code for all five services: traffic‑hijack, product‑admin, data‑migration, data‑quality, and product‑read. Core fields and business logic are trimmed but retain essential implementations such as the EAV schema, Loader mechanism, OHC integration, Redis cache design, MQ‑driven cache updates, interface‑level validation, and the responsibility‑chain for traffic hijacking.

Conclusion

The system has been validated in a production environment serving billions of users, handling 70 000 concurrent read requests on 32 machines, and surviving multiple large‑scale promotions. Each solution described is derived from real traffic, encountered problems, iterative fixes, and re‑validation, offering concrete, deployable designs rather than abstract concepts.