High-Concurrency System Performance Optimization and Testing Guide

Today we present a comprehensive article on high‑concurrency system architecture metrics and tuning experience, originally published on OpsDev and reproduced with author permission.

The two most important performance indicators are Throughput (tasks processed per unit time) and Latency (average response time per task). Higher throughput often increases server load and can raise latency, while lower latency usually allows higher throughput.

Performance testing follows three steps: define business scenarios to set acceptable latency, choose a load‑testing tool (Tsung, an open‑source distributed stress tester supporting HTTP, MySQL, MQTT, etc.), and iteratively increase load while monitoring server load and latency to find system limits.

General Linux kernel parameters must be adjusted for high concurrency. Edit /etc/sysctl.conf and add:

fs.nr_open = 100000000 fs.file-max = 100000000

These settings raise the maximum number of open files system‑wide. The nr_open and file-max meanings can be found in the kernel documentation. Note that /proc/sys/fs/inode-max may not exist on all systems.

For user‑level limits, modify /etc/security/limits.conf :

* hard nofile 4194304 * soft nofile 4194304

These increase the per‑process file descriptor limit, which is crucial because each TCP connection consumes one descriptor.

Client‑side tuning includes expanding the available port range. Add to /etc/sysctl.conf :

net.ipv4.ip_local_port_range = 1024 65535

If the same machine runs both server and client, reserve service ports to avoid conflicts:

net.ipv4.ip_local_reserved_ports = 5222, 5269, 5280-5390

Erlang/OTP services also need VM parameter adjustments. Increase the process limit:

$ erl +P 10000000

Raise the atom limit:

$ erl +t 10000000

And increase the port limit:

$ erl +Q 10000000

When tuning Tsung scripts, reduce TCP buffers to 4 KB:

<option name="tcp_snd_buffer" value="4096"/> <option name="tcp_rcv_buffer" value="4096"/>

Lower the idle think‑time to 5 seconds:

<option name="hibernate" value="5"/>

Disable heavy logging and traffic dumps to save I/O:

<tsung loglevel="error" dumptraffic="false" version="1.0">

Apply a token‑bucket rate limit for outbound traffic (KB/s):

<option name="rate_limit" value="1024"/>

To locate performance bottlenecks, use standard tools: top for CPU/memory, iotop or iostat for disk I/O, and iftop for network bandwidth. If issues persist, revisit system parameter tuning and check for single‑point code bottlenecks such as synchronized supervisor calls.

In summary, the article shares practical problems encountered during load testing and offers solutions, inviting readers to discuss and improve upon the presented methods.