Performance Root Cause Analysis: Decision Trees & Evidence Chains Explained

Why Decision Trees and Evidence Chains Matter

Some engineers locate a performance bottleneck in 10 minutes while others spend a day guessing. The difference is not experience but two maps: a performance analysis decision tree that tells you what to look at in the system, and a bottleneck evidence chain that walks you from symptom to root cause. These are the core of the RESAR seven-step method.

Decision Tree: A Three-Layer Static Map

The decision tree = all system components + each component's key modules + each module's counters, drawn as a structured tree. Three layers are essential:

Component layer – answers "what constitutes the system" (OS, database, middleware, network, etc.)

Module layer – answers "which blocks to examine per component" (e.g., for OS: CPU, memory, disk, network, swap)

Counter layer – answers "which metric measures each block"

The industry-standard USE method helps remember counters: for every resource check Utilization (U) , Saturation (S) , and Errors (E) . Apply to CPU, memory, disk, network and you have a complete tree.

Building the Decision Tree: Four Steps

List components – from the architecture diagram, enumerate OS, database, middleware, network, etc.

Refine modules – pick a component (e.g., OS) and break out CPU, memory, disk, network, swap.

List counters – capture every important first-level counter for each module; missing a key counter forces a re-run.

Draw correlations – single counters are often insufficient; map relationships between them to form diagnostic clues.

Start with only first-level counters; drill to second/third level only when direction is unclear. Listing everything upfront wastes time.

Reading Counters Correctly: CPU Example

top

shows 9 CPU counters: us, sy, ni, id, wa, hi, si, st plus load average. On a host, mpstat adds %guest and %gnice (11 total).

The most misused is load average . It is not simply "high means busy". Its precise meaning:

load average = vmstat proc column r + b , i.e., the 1/5/15-minute average of processes in runnable (r) + uninterruptible (b) state.

r

= running or waiting for CPU; b = waiting for I/O or memory swap (uninterruptible). Understanding this tells you whether high load points to CPU or I/O.

Monitoring tools are secondary; coverage of all counters is primary. If tools miss counters, supplement with commands. Do not blindly trust tools.

Evidence Chain in Action: From "Single Core 21.4%" to "Add Queues"

The evidence chain is the decision tree applied to a concrete case – a full record of the analysis logic. It proceeds in two phases:

1. Global Monitoring – Scan Overview Counters

Real case: a 24-core host. %us not high, %id still has headroom, but CPU 22's %si (softirq) spikes to 21.4% . One core handling all softirqs across 24 cores is clearly abnormal.

2. Targeted Monitoring – Drill Down Step by Step

Check /proc/softirqs → NET_RX (network receive) counter concentrated on CPU 22.

Inspect /sys/class/net/<nic>/queues/ → only 1 RX queue ; all packet processing funneled to one core.

Solution → increase queues to 8 (VM: edit KVM XML; bare metal: replace NIC), spreading interrupts across 8 cores.

Critical reminder: always start with global scan, then drill down. Jumping straight to cat /proc/softirqs is too targeted and risks misdirection.

One-Page Logic Summary

Decision tree = static map (what to examine)

Evidence chain = dynamic path (how to reach root cause)

Mnemonic for execution:

List components → Refine modules → List all counters → Draw correlations → Global scan phenomenon → Targeted drill root cause → Single-point tune → Full-chain regression verify

Understand your system and architecture first; otherwise all techniques are castles in the air.

Stop Guessing

Analysis without an evidence chain is fundamentally guessing – and guessing will be wrong. Draw these two maps, use them, and you can truly own performance results.

Save this article, apply the steps to your own system – doing it once beats reading ten times.