Designing High-Concurrency Systems: Lessons from a Sports Venue Management Platform

Project Background

Rapid digitalization forces traditional sports venues to replace manual operations with an integrated multi‑tenant e‑commerce platform that handles product sales, venue booking, smart lighting, and payment settlement. Peak traffic during flash‑sale events can exceed normal load by more than 20 times, demanding a robust high‑concurrency architecture.

Key Design Dimensions

1. Multi‑Level Cache Design

A three‑tier cache is employed: client‑side + CDN edge cache for static resources, application‑level local cache with an 85 % hit rate for hot data, and a Redis Cluster for shared state such as inventory, user sessions, and seckill eligibility. Bloom filters block requests for nonexistent product IDs, random TTL jitter mitigates cache avalanche, and mutex locks with logical expiration prevent cache breakdown. Inventory deductions use Redis DECR atomic operations to eliminate overselling.

2. Asynchronous Processing & Message Queue

RocketMQ decouples the synchronous request flow into an event‑driven model. A seckill request performs only a lightweight Redis pre‑deduction and qualification check, then publishes a success event to the queue. Consumers asynchronously persist orders, update inventory, and handle coupon redemption, smoothing write pressure and preventing database overload. RocketMQ’s transactional messages guarantee eventual consistency between local transactions and message delivery.

3. Database Optimization

Read‑write separation with a master‑slave MySQL cluster distributes reads (8:2 read‑write ratio). ShardingSphere horizontally shards large tables (e.g., orders, inventory) into 16 shards, each capped at 500 k rows, avoiding large‑table performance degradation. Indexes are refined via EXPLAIN, covering indexes eliminate full scans, and slow‑query logs are analyzed to optimize the top‑10 slow SQL statements. Connection‑pool parameters are tuned based on load‑test results.

4. Horizontal Scaling & Elasticity

All microservices run in Docker containers orchestrated by Kubernetes. Horizontal Pod Autoscaler (HPA) triggers scaling when CPU usage exceeds 70 % or custom QPS thresholds are crossed, expanding pods within 30 seconds. Pre‑emptive scaling before major promotions expands replica counts to anticipated peak levels, verified by stress testing.

Performance Problems and Solutions

Problem 1: Overselling and DB Saturation During Seckill

Initial QPS reached 1.8 × 10⁴, causing MySQL connection‑pool exhaustion and response times >8 s. Migrating inventory to Redis with atomic DECR and queuing order writes reduced QPS to ~800 during the peak, lowered CPU utilization from 98 % to 35 %, and increased seckill throughput from 2 k QPS to 16 k QPS, eliminating overselling.

Problem 2: Cache Breakdown for Hot Products

During a promotion, a popular product’s cache expired simultaneously, flooding the database and raising response times to 1200 ms. Introducing logical expiration with mutex lock and Sentinel‑based circuit breaking (timeout > 300 ms) raised cache hit rate from 76 % to 99.2 %, cut database load by ~90 %, and restored response time to 150 ms.

Problem 3: Multi‑Tenant Resource Contention

Large tenants monopolized thread‑pool resources, causing small tenants’ P99 latency to rise from 3.5 s to 400 ms after introducing tenant‑level thread‑pool isolation and per‑tenant QPS quotas. Service quality became balanced across tenants.

Results and Reflection

After applying the six‑dimensional solution, average interface latency dropped from 1200 ms to 150 ms, throughput increased eightfold, and overall availability reached 99.95 %, supporting stable operation during major sales events. The author notes that high‑concurrency design is a systematic engineering effort requiring continuous load testing and monitoring.

Future Work

Planned enhancements include adopting a Service Mesh for finer‑grained traffic governance and exploring stronger read‑write cache consistency models to further improve resilience and latency.