Designing Three‑High Systems: Practical Performance Tuning and Fault‑Tolerant Architecture

In today’s distributed environment, the "three‑high" goals—high performance, high concurrency, and high availability—are fundamental for handling traffic spikes and ensuring stable operation.

High Performance: Starting Point and Core

Performance is deemed the most critical because it directly determines processing speed and throughput; higher performance naturally supports higher concurrency and reduces latency‑related availability issues.

Typical bottlenecks appear in three layers:

Compute layer : complex business logic, frequent Full GC, inefficient algorithms.

Communication layer : slow downstream services, high network latency, low‑efficiency serialization.

Storage layer : large tables, slow SQL, poor index design, or mis‑configured ES shards.

Read Optimization: Making Queries Faster and More Efficient

The core idea is to reduce direct requests to the storage layer. Six common techniques are:

Cache : Use a bypass‑cache pattern—query cache first, fall back to DB on miss, and write‑back. For rarely‑changed data (e.g., user‑level rules), add local cache to shrink response time from milliseconds to microseconds.

Parallelism : Convert serial I/O into multi‑threaded execution. For an order‑placement API that sequentially queries user info, inventory, and coupon status (each 100 ms), parallelism can cut total latency to ~100 ms.

Batching : Replace many single‑row queries with a single batch query. When fetching 100 users, a batch reduces 100 round‑trips to one.

Data compression : Compress large payloads (e.g., gzip or Snappy) and return only necessary fields to lower bandwidth.

Read‑write separation : Use master‑slave replication; writes go to the master, reads are served by replicas, preventing write locks from blocking reads.

Pooling : Reuse expensive resources such as threads or connections via pools, reducing creation overhead.

Write Optimization: Making Storage More Stable and Fluid

Write bottlenecks affect system stability, especially during peak traffic. Four main techniques are:

Asynchrony : Buffer write requests in a message queue, return success immediately, and let consumer threads persist data later (e.g., order info written to MQ then stored asynchronously).

Batching : Group multiple inserts into a single transaction, dramatically reducing commit and log‑flush overhead (e.g., inserting 1 000 orders in one batch versus 1 000 single inserts).

Lock‑free design : Reduce lock granularity, such as splitting a global balance lock into per‑account sub‑locks, or replacing locks with FIFO queues in extreme cases.

Data sharding : Distribute writes across multiple shards (e.g., hash‑based user‑ID sharding) and separate hot and cold data to improve write throughput.

High Concurrency: Scaling the Cluster

Beyond single‑node performance, concurrency requires cluster‑level expansion, described as three‑dimensional scaling: horizontal, vertical, and “depth” scaling.

Horizontal Scaling

Increasing node count is the most common approach.

Application layer : Stateless services are containerized and scaled (e.g., expanding order service from 10 to 50 instances for a 5× concurrency boost).

Storage layer : Add shards and migrate data (e.g., expanding an ES cluster from 3 to 6 shards).

Horizontal scaling is simple but storage expansion can be costly.

Vertical Scaling

Decompose a monolith into microservices, allowing independent scaling of business domains.

Example: split an e‑commerce platform into user, product, order, and payment services; during a promotion, only order and payment services need scaling.

Vertical scaling also involves database sharding and unitization:

Sharding : Separate databases by business (order DB, user DB) and tables by dimension (time‑based order tables), adding more DB instances to raise connection limits.

Unitization : Partition system and data by region (e.g., Guangdong unit serves Guangdong users), creating isolated traffic loops that avoid single‑datacenter bottlenecks.

This approach improves both concurrency and availability because a failure in one unit does not affect others.

High Availability: Fault‑Tolerant Design

Microservice decomposition introduces new failure modes (network glitches, node crashes). Fault tolerance aims to protect the system during failures and keep business running.

Two core capabilities:

Protection : Prevent fault propagation (e.g., isolate a crashed service to avoid thread blockage).

Recovery : Redirect requests to healthy nodes via failover or retries.

Practical Fault‑Tolerance Strategies

Failover : On node failure, retry other nodes. Suitable for most non‑idempotent scenarios; limit retry attempts.

Recovery : Record failed requests and retry later; requires idempotency (e.g., message notifications).

Fast Fail : Immediately return error without retry for non‑idempotent operations like payments.

Silent Fail : Mark a faulty node and stop routing to it for a period; set appropriate expiration.

Safe Fail : Ignore non‑critical errors; succeed if core business succeeds.

Parallel Call : Invoke multiple nodes concurrently, return on first success; use only for highly time‑critical paths and ensure idempotency.

Broadcast Call : Call all nodes and require all to succeed; reserved for strong consistency needs despite performance cost.

Core Fault‑Tolerance Patterns

Circuit Breaker : When error rate exceeds a threshold, open the circuit to reject calls, then half‑open after a cooldown to test recovery (e.g., payment service error rate >50% triggers circuit break).

Retry : Retry transient network failures up to three times, only for idempotent requests, and ensure total retry time stays within upstream timeout.

Bulkhead (Compartment Isolation) : Isolate resources per business (e.g., separate thread pools for order and SMS services) so a failure in one does not affect the other.

Conclusion: Core Logic of Three‑High Design

High performance forms the foundation; read/write optimizations make a single node efficient.

High concurrency extends capacity through horizontal, vertical, and depth scaling.

High availability safeguards the system with fault‑tolerant designs.

In practice, choose solutions based on business scenarios: prioritize caching and read‑write separation for C‑end APIs, apply asynchrony and horizontal scaling for promotion spikes, and enforce fault‑tolerance and isolation for core transactions.