Mastering High-Concurrency System Design: 18 Essential Techniques

Hello everyone, I'm Su San, back again.

Preface: A fan recently asked how to design a high‑concurrency system, a frequent interview topic that tests both breadth and depth of technology.

This article discusses key points for high‑concurrency system design.

1 Page Staticization

For high‑concurrency page functionality, we must implement

static

design. Rendering pages dynamically for massive concurrent users overloads the server.

We can use template engines like

Freemarker

or

Velocity

to generate static pages.

For example, a job can periodically fetch homepage data, render it with a template engine into an HTML file, and then use a

shell

script to sync the file to the front‑end servers.

2 CDN Acceleration

Static pages improve speed, but users are geographically distributed. A CDN (Content Delivery Network) delivers content from nodes close to users, reducing latency and increasing hit rates.

CDN copies static assets (images, CSS, JS) to global nodes and serves them based on user location.

Common Chinese CDN providers include Alibaba Cloud CDN, Tencent Cloud CDN, and Baidu Cloud Acceleration.

3 Caching

Caching is essential in high‑concurrency systems.

Application‑server memory cache (second‑level cache).

Distributed cache middleware such as Redis or Memcached.

Second‑level cache offers better performance but may cause data inconsistency across multiple servers. Distributed caches avoid this issue but are slightly slower.

Typical usage involves caching product categories, reducing database load and improving performance.

Be aware of consistency problems, cache penetration, breakdown, and avalanche.

4 Asynchronous Processing

Not all interface logic needs to be synchronous. Core business operations can be synchronous, while non‑core tasks (notifications, logging) can be processed asynchronously.

4.1 Thread Pool

After refactoring with a thread pool, non‑core tasks are submitted to separate thread pools, improving interface performance. However, if the server restarts or a task fails, data may be lost.

4.2 MQ

Using a message queue, the interface only sends MQ messages; consumers execute the actual tasks, providing fast response and reliability.

5 Multi‑Threaded Processing

When massive MQ messages accumulate, using a thread pool to consume them can alleviate backlog. Core and max thread counts, queue size, and idle timeout should be configurable.

Note: Multi‑threaded consumption may affect message order; choose appropriate solutions for ordered processing.

6 Database Sharding

When database throughput becomes a bottleneck, sharding (horizontal) and vertical splitting can distribute load.

Horizontal sharding splits tables by ID modulo, range, or consistent hashing. Vertical splitting separates business domains.

Vertical splitting solves connection and I/O limits; horizontal splitting addresses large table scans.

7 Pooling Techniques

Connection pools (e.g., Druid, C3P0, Hikari, DBCP) reuse database connections, reducing creation overhead.

8 Read‑Write Separation

Following the 80/20 rule, most traffic is reads. Using master‑slave replication, writes go to the master, reads are served by slaves, improving scalability.

For larger traffic, a master‑multiple‑slave architecture distributes reads further.

9 Index Optimization

Indexes accelerate queries on large tables but add overhead on inserts. Optimize by creating composite indexes, dropping unused indexes, using

EXPLAIN

, and handling index invalidation.

10 Batch Processing

Instead of querying each user individually, batch queries retrieve multiple users in a single DB call.

public List<User> queryUser(List<User> searchList) {
    if (CollectionUtils.isEmpty(searchList)) {
        return Collections.emptyList();
    }
    List<Long> ids = searchList.stream().map(User::getId).collect(Collectors.toList());
    return userMapper.getUserByIds(ids);
}

11 Clustering

Deploying multiple server nodes forms a cluster to ensure high availability. For Redis, a three‑master cluster distributes data across nodes, with each master having a slave for failover.

12 Load Balancing

Load balancers (Nginx, LVS, Haproxy, F5) distribute requests across servers using strategies like round‑robin, weight, IP hash, least connections, and shortest response time.

13 Rate Limiting

To protect stability, limit requests per user, per IP, or per interface using Nginx or Redis. Captchas (including sliding captchas) provide precise control.

14 Service Degradation

During overload, non‑core features can be disabled via configuration switches (e.g., Apollo). Fallback data can be provided when primary sources fail. Hystrix and Sentinel are common circuit‑breaker tools.

15 Failover

When a server becomes unresponsive, failover mechanisms automatically route traffic to healthy nodes, using Ribbon for load balancing and Hystrix for circuit breaking.

16 Multi‑Active Deployment

Deploying the system in multiple data centers (e.g., Shenzhen, Tianjin, Chengdu) ensures continuity. Traffic is routed via DNS and routing servers, with data synchronization to maintain consistency.

17 Stress Testing

Before launch, estimate QPS, perform load testing (JMeter, Locust, PTS), and provision sufficient server capacity (often 3× estimated load).

18 Monitoring

Use Prometheus to monitor metrics such as response time, third‑party latency, slow SQL, CPU, memory, disk, and database usage, enabling timely alerts and troubleshooting.