High‑Availability Architecture and Optimization Strategies for a Large‑Scale Membership System

The membership system is a core service that supports order processing across all business lines, requiring high performance and availability because any failure blocks user orders company‑wide.

ES High‑Availability Solution

A dual‑center primary‑backup Elasticsearch cluster is deployed across two data centers (A and B). The primary cluster handles reads/writes, while data is asynchronously replicated to the backup via MQ. In case of a primary outage, traffic is switched to the backup with minimal downtime.

Traffic Isolation with Three‑Cluster ES Architecture

To protect the primary cluster from traffic spikes caused by marketing campaigns, a separate ES cluster handles high‑TPS marketing requests, isolating them from the critical order‑flow cluster.

Deep ES Optimizations

Improvements include balancing shard distribution, tuning thread‑pool sizes, limiting shard memory to 50 GB, removing unnecessary text fields, using filters instead of queries, performing sorting in the application layer, and adding routing keys to reduce cross‑shard requests.

Member Data Caching with Redis

A dual‑center Redis setup writes to both data centers simultaneously and reads locally, providing high availability and reducing latency. A distributed lock ensures cache consistency with near‑real‑time Elasticsearch updates.

MySQL Dual‑Center Partition Cluster

The legacy SQL Server store is migrated to a sharded MySQL cluster (1000+ shards, 1‑master‑3‑slave per data center) with sub‑millisecond replication, supporting seamless read/write routing via DBRoute.

Zero‑Downtime Migration Strategy

The migration uses full data sync, incremental sync, and real‑time dual‑write. During a trial phase, writes go to SQL Server while asynchronous writes to MySQL are retried on failure; once stable, traffic is gradually shifted to MySQL using A/B testing and result verification.

Fallback to ES as Primary Store

If the DAL component or MySQL fails, reads/writes can be switched to Elasticsearch, with later synchronization back to MySQL once the issue is resolved.

Abnormal Account Governance

Complex logic identifies and fixes accounts that mistakenly bind multiple user identities, preventing cross‑account data leakage and order manipulation.

Refined Flow‑Control and Degradation Strategies

Global, hotspot, and caller‑based rate‑limiting protect the system from overload; degradation triggers on high average response times or error ratios, automatically circuit‑breaking problematic paths.