Performance Optimization Basics: Metrics, APM Tools, Bottleneck Analysis for Backend

Performance Metric System

Response Time

Response time measures the duration from request initiation to response receipt. Typical latency thresholds are:

Local cache hit: < 1ms (excellent), < 5ms (good), < 10ms (acceptable), >10ms (needs optimization)

Memory access: < 0.1ms (excellent), < 1ms (good), < 10ms (acceptable), >10ms (needs optimization)

SSD read: < 1ms (excellent), < 5ms (good), < 10ms (acceptable), >10ms (needs optimization)

Remote cache (Redis): < 1ms (excellent), < 5ms (good), < 10ms (acceptable), >10ms (needs optimization)

Database query: < 10ms (excellent), < 50ms (good), < 200ms (acceptable), >200ms (needs optimization)

Remote service call: < 50ms (excellent), < 200ms (good), < 500ms (acceptable), >500ms (needs optimization)

Throughput

QPS (Queries Per Second) – core read‑performance metric. Example Java implementation uses AtomicLong and System.currentTimeMillis() to count requests and compute QPS every second.

TPS (Transactions Per Second) – measures overall processing capacity. Example Java class TPSMonitor stores timestamps in a ConcurrentLinkedQueue<Long> and computes the count within a 1000 ms sliding window.

Resource Utilization

CPU utilization example uses OperatingSystemMXBean to print core count and system load, and the Sigar library ( CpuPerc) to report user, system, wait, and idle percentages.

Memory utilization example retrieves heap and non‑heap statistics via Runtime and MemoryMXBean, formatting bytes to human‑readable units.

Google SRE Golden Metrics

Four golden metrics:

Latency – P50/P90/P99, slow‑request ratio.

Traffic – QPS/TPS, concurrent connections, bandwidth.

Errors – HTTP 5xx rate, timeout rate, business error rate.

Saturation – CPU/memory usage, queue depth, disk usage.

Bottleneck Analysis

Identification Methodology

CPU bottleneck: use async-profiler with command

java -jar async-profiler.jar start -e cpu -f profile.html <pid>

or Java Flight Recorder (JFR). Example JFR monitor attaches to the VM, starts a recording ( id=1,filename=recording.jfr,duration=60s), and exports the file from /tmp/recording.jfr.

I/O bottleneck: commands iostat -x 1, pidstat -d 1, iotop -o. Sample iostat output shows device reads/writes, await time and %util.

Network bottleneck:

netstat -an | awk '/^tcp/ {++s[$NF]} END {for(a in s) print a, s[a]}'

, nload -t 1000, tcpdump -i eth0 -w traffic.pcap, and ss -s for socket statistics.

Flame Graph Generation

perf:

perf record -F 99 -a -g -- sleep 60
perf script | ./stackcollapse-perf.pl | ./flamegraph.pl > perf.svg

async‑profiler: ./profiler.sh -d 60 -f profile.svg <pid> go‑torch (Go): go-torch -b profile.out Pyroscope (continuous profiling) Docker compose snippet:

version: '3'
services:
  pyroscope:
    image: pyroscope/pyroscope:latest
    ports:
      - "4040:4040"
    environment:
      - PYROSCOPE_SERVER_ADDRESS=http://pyroscope:4040

Diagnostic Tool Comparison

jstat – JDK built‑in, GC statistics, low overhead.

jstack – JDK built‑in, thread dump, low overhead.

jmap – JDK built‑in, heap memory, medium overhead.

arthas – Alibaba open‑source, diagnostic tool, medium overhead.

async‑profiler – Open source, CPU/Memory analysis, low overhead.

perf – Linux system‑level analysis, low overhead.

BCC/bpftrace – Linux kernel tracing, controllable overhead.

honest‑profiler – Open source, very low overhead profiler.

APM Toolchain

SkyWalking

Distributed tracing system. Docker‑compose defines Elasticsearch, OAP server, and UI services. Java agent configuration (service name, collector address, sampling rate) is placed in skywalking-agent.yml. Spring Boot integration adds Maven dependency org.apache.skywalking:apm-toolkit-trace:9.1.0 and annotates methods with @Trace, using ActiveSpan.tag("user_id", id.toString()) to attach custom tags.

Arthas

Start with java -jar arthas-boot.jar. Common commands: dashboard – system overview. thread -n 5 – top 5 busy threads. trace com.example.Service 'cost>100' – trace methods costing >100 ms.

watch com.example.Service methodName "{params,returnObj}" -x 2

– watch parameters and return values. jad com.example.Service – decompile class. profiler -d 30 -f profile.html – CPU profiling. logger -c com.example.Service --name ROOT --level DEBUG – change log level at runtime.

Example watch output shows method arguments and the returned User(id=1, name="张三", email="[email protected]").

Link Tracing Integration

Spring Cloud Sleuth + Zipkin Maven dependencies and application.yml with Zipkin base URL and sampler probability 0.1 (10 %). Manual trace propagation uses Tracer tracer to create a span, tag it with span.tag("user_id", id.toString()), execute business logic, and close the span.

Optimization Methodology

Principles

1. Measure first. 2. Identify bottlenecks (80/20 rule). 3. Quantify goals. 4. Apply small incremental changes. 5. Verify effect by comparing before/after data. 6. Run regression tests. Benefit formula:

(pre‑opt time - post‑opt time) × request volume × frequency

.

Dimensions

Network – DNS cache, HTTP/2, connection pooling.

Access – Load balancing, SSL termination (Nginx, HAProxy).

Application – Cache, asynchronous processing, batch handling.

Data – Indexes, SQL tuning, sharding.

Infrastructure – Horizontal scaling, hardware upgrade, read/write split.

Checklist Code

public class PerformanceChecklist {
    // 1. Database layer
    public void checkDatabase() {
        // [ ] Are slow queries indexed?
        // [ ] Is an appropriate connection pool (e.g., HikariCP) used?
        // [ ] Is the N+1 query problem avoided?
        // [ ] Is pagination efficient?
        // [ ] Are batch operations using batch processing?
    }
    // 2. Cache layer
    public void checkCache() {
        // [ ] Is hot data cached?
        // [ ] Are cache keys well designed?
        // [ ] Is cache penetration handled?
        // [ ] Is cache avalanche handled?
        // [ ] Is multi‑level caching used?
    }
    // 3. JVM layer
    public void checkJVM() {
        // [ ] Is heap size reasonable?
        // [ ] Is GC frequency normal?
        // [ ] Are there memory leaks?
        // [ ] Is thread‑pool configuration reasonable?
    }
    // 4. Code layer
    public void checkCode() {
        // [ ] Are efficient data structures used?
        // [ ] Are unnecessary object creations avoided?
        // [ ] Is async processing used where appropriate?
        // [ ] Are there duplicate calculations inside loops?
    }
}

Performance Testing

Tool Comparison

JMeter – full‑featured GUI, suited for complex functional scenarios.

Gatling – Scala‑based, beautiful reports, suited for performance testing and CI integration.

wrk – lightweight, high‑performance, suited for quick benchmarks.

ab – simple load testing.

k6 – JavaScript scripting, cloud‑native.

Locust – Python scripting, distributed user‑level simulation.

JMeter Example

<jmeterTestPlan version="1.2" properties="5.0">
  <hashTree>
    <TestPlan>
      <stringProp name="TestPlan.comments">API Performance Test</stringProp>
      <stringProp name="TestPlan.threads">100</stringProp>
      <stringProp name="TestPlan.rampTime">60</stringProp>
      <stringProp name="TestPlan.duration">300</stringProp>
    </TestPlan>
    <hashTree>
      <HTTPSamplerProxy guiclass="HttpTestSampleBean" testclass="HTTPSamplerProxy">
        <stringProp name="HTTPSampler.domain">api.example.com</stringProp>
        <stringProp name="HTTPSampler.port">8080</stringProp>
        <stringProp name="HTTPSampler.path">/api/users</stringProp>
        <stringProp name="HTTPSampler.method">GET</stringProp>
      </HTTPSamplerProxy>
    </hashTree>
  </hashTree>
</jmeterTestPlan>

Gatling Example

import io.gatling.core.Predef._
import io.gatling.http.Predef._
import scala.concurrent.duration._

class ApiSimulation extends Simulation {
  val httpConf = http
    .baseUrl("http://api.example.com")
    .acceptHeader("application/json")
    .contentTypeHeader("application/json")

  val getUsers = scenario("获取用户列表")
    .exec(http("get_users").get("/api/users").check(status.is(200)))

  val createUser = scenario("创建用户")
    .exec(http("create_user").post("/api/users")
      .body(StringBody("""{"name":"test","email":"[email protected]"}""")
      .check(status.is(201)))

  val mixedScenario = scenario("混合场景")
    .exec(getUsers)
    .pause(1)
    .exec(createUser)

  setUp(
    mixedScenario.inject(
      atOnceUsers(100), // 瞬时 100 用户
      rampUsers(500).during(60) // 60秒内爬坡到 500 用户
    )
  ).protocols(httpConf)
   .assertions(
     global.responseTime.percentile(95).lt(200),
     global.successfulRequests.percent.gt(95)
   )
}

wrk Example

# Basic load test
wrk -t12 -c400 -d30s http://api.example.com/api/users

# Advanced usage with latency, timeout, headers, and Lua script
wrk -t12 -c400 -d30s \
  --latency \
  --timeout 5s \
  -H "Authorization: Bearer token" \
  -H "Content-Type: application/json" \
  -s post.lua \
  http://api.example.com/api/users

# post.lua
wrk.method = "POST"
wrk.body   = '{"name":"test"}'
wrk.headers["Content-Type"] = "application/json"

Performance Test Process

Performance Test Lifecycle
1. Requirement Analysis
   - Define performance goals (QPS, RT, concurrency)
   - Scope definition
   - Success criteria
2. Test Design
   - Scenario design (normal, peak, error)
   - Data preparation
   - Script development
3. Test Execution
   - Warm‑up
   - Baseline test
   - Load test
   - Stress test
4. Result Analysis
   - Collect metrics
   - Locate bottlenecks
   - Provide optimization suggestions
5. Optimization Verification
   - Apply performance improvements
   - Regression validation