Tuning C‑states, P‑states and CPU Frequency for Low‑Latency High‑Frequency Trading

Who Is Affected

Busy‑poll strategy cores never enter idle, so C‑state tuning does not apply to them. The remaining cores that matter are (1) non‑isolated housekeeping cores that handle NIC interrupts, (2) threads that block on I/O (risk, reconciliation, downstream sending), and (3) package‑level states that only become active when all cores are idle.

C‑state: Meaning and Cost

C‑state describes how deep a core sleeps when idle. Higher numbers mean deeper sleep, lower power, but longer exit latency. Two key metrics are exposed in /sys: exit latency (time to execute the first instruction after wake‑up) and target residency (minimum time the core must stay in that state for the power saving to be worthwhile).

# Example to list latency and residency for CPU3
cd /sys/devices/system/cpu/cpu3/cpuidle
for s in state*/; do echo "$(cat $s/name) latency=$(cat $s/latency)us residency=$(cat $s/residency)us"; done

On Skylake‑SP, C1 has 2 µs latency, C1E 10 µs, and C6 133 µs (package‑wide worst‑case). Deeper states increase wake‑up time and may also flush caches and TLBs, so the real cost can be higher than the documented latency.

Who Decides the State

The kernel driver (e.g., intel_idle or acpi_idle) enumerates supported states. The governor (menu, teo, ladder, haltpoll) selects a state based on predicted wake‑up time. Because a mis‑prediction can add hundreds of microseconds, the recommended approach is to remove deep states from the candidate set so the governor can only pick shallow ones.

Four Ways to Restrict C‑state

1. Kernel boot parameters

intel_idle.max_cstate=1 processor.max_cstate=1

These parameters drop any state with an index greater than 1 (i.e., C6) from the enumeration. The same syntax works for both Intel ( intel_idle) and AMD ( acpi_idle) drivers.

2. idle=poll (or idle=halt )

Adding idle=poll disables the entire cpuidle subsystem, forcing idle cores to spin in a lightweight loop and never enter any C‑state. This eliminates wake‑up latency but prevents high‑frequency P‑states that require at least one idle core, so it can reduce peak frequency on some workloads.

3. PM‑QoS per‑core latency limit

Write a maximum wake‑up latency (in µs) to /dev/cpu_dma_latency for a global limit, or to

/sys/devices/system/cpu/cpu<N>/power/pm_qos_resume_latency_us

for a per‑core limit. The file descriptor must stay open for the constraint to remain active.

# Global limit (keep a process alive)
echo 0 > /dev/cpu_dma_latency
# Per‑core limit for CPUs 0‑1 (allow only C1)
for c in 0 1; do echo 2 | sudo tee /sys/devices/system/cpu/cpu${c}/power/pm_qos_resume_latency_us; done

4. Disable a specific state

# Disable state2 (assumed to be C6) on CPU3
echo 1 | sudo tee /sys/devices/system/cpu/cpu3/cpuidle/state2/disable

This is the most granular method but requires knowing the exact state index.

P‑state and Frequency Scaling

P‑states control how fast a core runs when active. P0 is the highest (Turbo), P1 is the base frequency, and higher numbers are lower frequencies. The kernel’s cpufreq subsystem has three layers: core framework, governor, and driver.

Common governors

performance : always request the highest allowed frequency.

powersave : always request the lowest frequency.

schedutil : follows scheduler utilization (PELT).

ondemand , conservative : sample idle time and adjust.

For low‑latency workloads, performance is recommended because busy‑poll cores already stay at 100 % utilization, and other cores benefit from a fixed high frequency without the governor’s sampling delay.

# Set performance governor for all CPUs
sudo cpupower frequency-set -g performance
cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor

Intel intel_pstate modes

The default active + HWP mode lets the hardware pick P‑states. Setting the governor to performance forces the driver to pin the frequency range to the maximum and zero out energy‑performance preferences. In active (no HWP) the driver still follows the scheduler, while passive falls back to the generic cpufreq driver.

Turbo control

Writing 1 to /sys/devices/system/cpu/intel_pstate/no_turbo disables Turbo, fixing the frequency at the base P1 level. This removes the variability caused by Turbo’s dependence on idle cores and power budget, which can be detrimental to p99.9 latency targets.

# Disable Turbo
sudo sh -c "echo 1 > /sys/devices/system/cpu/intel_pstate/no_turbo"

Practical AWS Considerations

Not all EC2 instance types expose C‑state or P‑state control to the guest OS. Large‑scale instances (e.g., c4, m4, r4, x1, i3, p3) and all bare‑metal instances give full control. Newer Nitro instances often expose only C‑state, while Graviton instances expose neither.

Before applying any tuning, verify control with:

# Check cpuidle and cpufreq visibility
ls -d /sys/devices/system/cpu/cpu0/cpuidle
ls -d /sys/devices/system/cpu/cpu0/cpufreq
cat /sys/devices/system/cpu/cpuidle/current_driver || echo "no cpuidle driver"
cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_driver || echo "no cpufreq driver"

If the directories or drivers are missing, the instance does not allow those knobs and any boot‑parameter changes will be ignored.

Applying boot parameters on different distros

On Amazon Linux edit /etc/default/grub and run grub2-mkconfig -o /boot/grub2/grub.cfg. On RHEL/CentOS 8+ use grubby --update-kernel=ALL --args="…". On Ubuntu add a file under /etc/default/grub.d/ that appends to GRUB_CMDLINE_LINUX_DEFAULT and run update-grub.

Verification Checklist

After reboot, confirm the configuration:

Driver and governor: cat /sys/devices/system/cpu/cpuidle/current_driver and cat /sys/devices/system/cpu/cpuidle/current_governor.

Available idle states via cpupower idle-info (deep states should be absent).

Usage counters for each state (disabled states show zero usage).

Active frequency governor and Turbo status.

Measure real frequencies with turbostat (check Bzy_MHz for busy cores).

Per‑core idle percentages with cpupower monitor.

Conclusion

Effective low‑latency tuning requires first confirming that the guest OS has control over C‑states and P‑states, then applying the appropriate kernel parameters, PM‑QoS limits, or governor settings, and finally validating the effect with cpupower and turbostat. On cloud instances without control, the only viable path is to move to bare‑metal or accept the host‑managed power policy.