How to Measure and Optimize System Load Capacity for High‑Concurrency Backends

In the internet era, high concurrency is a critical performance indicator for both web sites and app backends. The maximum number of concurrent requests a system can handle defines its load capacity, exemplified by Alibaba’s ability to sustain hundreds of millions of requests during Double 11.

1. Measurement Metrics

The primary metric is Requests per Second (RPS), which counts only successfully responded requests. As concurrency increases, RPS rises until a tipping point where additional users cause RPS to drop and response time to increase; this point marks the system’s maximum load capacity.

2. Influencing Factors

Key factors include:

Bandwidth

Hardware configuration (CPU, memory, storage)

System configuration (file descriptor limits, process/thread limits, TCP kernel parameters)

Application server configuration

Program logic

Overall system architecture

Bandwidth and hardware set the upper bound, while the other factors determine how close the system can approach that bound.

2.1 Bandwidth

Bandwidth (Mbps) limits the data transmission rate, analogous to pipe diameter.

2.2 Hardware Configuration

CPU frequency and core count affect computation speed and thread scheduling.

Memory size and speed influence data residency and access speed.

Disk speed (SSD vs. HDD) impacts I/O latency.

2.3 System Configuration (Linux)

File descriptor limits: /proc/sys/fs/file-max and per‑process ulimit -n.

Process/thread limits: ulimit -u, /etc/security/limits.conf.

TCP kernel parameters in /etc/sysctl.conf (e.g., net.ipv4.tcp_syncookies, net.ipv4.tcp_tw_reuse, net.ipv4.tcp_tw_recycle, net.ipv4.tcp_fin_timeout, etc.).

Example commands to view TCP connection states:

netstat -n | awk '/^tcp/ {++S[$NF]} END {for(a in S) print a, S[a]}'

Typical output shows many TIME_WAIT sockets, which should be reduced via kernel tuning.

2.4 Application Server Configuration

Common concurrency models:

Multi‑process (one process per request)

Prefork (process pool)

Worker (one thread per request)

Master/worker (event‑driven, non‑blocking I/O, used by Nginx)

For Nginx/Tengine, adjust worker count, keepalive timeout, and worker_rlimit_nofile. For Tomcat, tune JVM heap ( -Xms, -Xmx, -Xmn), thread stack size, and connector settings (protocol, maxThreads, connectionTimeout, acceptCount, maxConnection).

2.5 Database Layer

MySQL performance depends on hardware, configuration ( max_connections), schema design, query efficiency, and storage engine choice. When tables reach millions of rows, consider vertical or horizontal sharding and master/slave read‑write separation.

Redis, as an in‑memory single‑threaded cache, can offload hot data from MySQL. Deploy Redis cluster, Twemproxy, or Codis for high availability, noting each solution’s trade‑offs.

2.6 System Architecture

Typical high‑concurrency architecture: load balancer (hardware or software like LVS), reverse proxy (Nginx), application servers (Tomcat), databases (MySQL/DDL), and cache (Redis/Codis). Load balancing can be hardware (e.g., F5) or software (LVS, HAProxy). Software load balancers support layer‑4 (LVS) and layer‑7 (Nginx) routing.

3. General Architecture Example

A common Java backend stack is illustrated as:

By systematically evaluating each factor—bandwidth, hardware, OS limits, TCP settings, server and database configurations—engineers can identify bottlenecks and apply targeted optimizations to raise the system’s RPS ceiling.