Identify and Fix System Performance Bottlenecks: Key Metrics and Optimization

1. Common System Bottlenecks

Typical performance constraints include:

CPU : Continuous heavy computation can keep CPU usage high, leading to slow response, frequent Full GC, and context‑switch overhead. Keeping CPU utilization below 75% is generally advisable.

Memory : Java objects reside in the JVM heap; limited heap size can cause out‑of‑memory errors or leaks when it fills up.

Disk I/O : Although SSDs are faster than HDDs, disk read/write speeds remain far slower than memory, creating I/O bottlenecks for heavy data access.

Network : Bandwidth limits become critical as concurrent requests increase, potentially turning the network into a choke point.

Exceptions : Frequent exception handling in high‑concurrency scenarios adds overhead and degrades throughput.

Database : Database operations involve disk I/O; intensive reads/writes can saturate I/O and increase latency.

2. Key Performance Metrics

To evaluate system health, monitor the following indicators:

RT (Response Time) : Includes database response time, server‑side processing time (e.g., Nginx dispatch + application execution), network transmission time, and client‑side latency.

TPS (Throughput) : Measured as IOPS for disk (random read/write performance) and network throughput (maximum data rate without packet loss). Both depend on CPU, NIC, firewall, and I/O subsystems.

Resource Utilization :

CPU usage – check with vmstat, mpstat, or top.

Memory usage – inspect via free -m, vmstat, or top.

Disk I/O – monitor with iostat or iotop.

Network I/O – use netstat, ifconfig, or tcpstat.

3. Performance‑Testing Pitfalls

During load testing, the first runs may be slower because Java bytecode is interpreted before the JIT compiler identifies hot spots and compiles them to native code. Subsequent runs benefit from this compilation, appearing faster. Variability can also arise from background processes, network jitter, and differing GC behavior; averaging multiple runs helps mitigate these fluctuations.

4. Bottom‑Up Diagnosis Strategy

After a stress test, collect a report containing RT, TPS, TP99, CPU, memory, I/O, network, and JVM GC statistics. Then investigate in the following order:

Operating‑system layer – examine CPU, memory, disk, and network usage; check system logs for anomalies.

JVM layer – analyze GC frequency and memory allocation; review GC logs for signs of excessive collection.

Application layer – look for Java‑level issues such as inefficient code, excessive locking, or database query bottlenecks.

5. Top‑Down Optimization Approach

Optimization proceeds from business logic down to the underlying system:

Application‑level tuning

Code optimization to eliminate memory leaks and reduce Full GC.

Design improvements, e.g., using proxy patterns to share frequently created objects.

Algorithm selection to lower time complexity.

Middleware tuning (MySQL)

Table and index design – plan for horizontal/vertical scaling, choose appropriate data types, and implement sharding.

SQL optimization – use EXPLAIN to verify index usage and profile query execution.

MySQL configuration – adjust connection limits and cache sizes (key buffer, query cache, sort buffer).

Hardware and OS – disable swap, add RAM, upgrade to SSDs, and tune kernel parameters.

System tuning

OS kernel parameters – modify sysctl settings for networking and I/O.

JVM tuning – allocate suitable heap sizes and select appropriate GC algorithms; place long‑lived objects in the old generation to reduce young‑gen collections.

6. Optimization Strategies

Two classic trade‑offs guide decisions:

Time for space : When query speed is less critical, allocate more storage to reduce processing time.

Space for time : Use techniques like MySQL sharding to split large tables, thereby improving query performance.

7. Fallback Measures

If performance issues persist after tuning, apply safety nets:

Rate limiting at entry points with circuit‑breaker logic to reject excess traffic.

Horizontal scaling – automatically add service instances when load exceeds predefined thresholds.