Low‑Latency HFT: Isolating CPUs with isolcpus, nohz_full, rcu_nocbs and IRQ Affinity

1. Interference that remains after binding a thread to a CPU

Even after binding a strategy thread to a specific core, four classes of work can still run on that core:

Other runnable tasks : The affinity mask is one‑way; any process or kernel thread without an explicit mask (e.g., kworker) may be scheduled on the isolated CPU.

Timer interrupts (tick) : The kernel generates local timer interrupts at CONFIG_HZ (typically 250 Hz or 1 kHz). Each tick performs scheduler accounting, timer‑wheel advancement and RCU state‑machine updates, causing a context switch and cache pollution.

RCU callbacks : After an RCU grace period, callbacks (often memory frees) run in soft‑interrupt context on the local CPU; a batch can take tens to hundreds of microseconds.

Hardware interrupts : NIC packet reception, NVMe completion queues, etc., trigger IRQs that by default are distributed by irqbalance, so even an isolated core receives them.

These interferences split into asynchronous noise (interrupts, timers, scheduler preemption) that can be disabled via kernel configuration, and synchronous noise (system calls, page faults) that must be eliminated in the application code.

2. isolcpus : Removing the core from the scheduler domain

isolcpus

is a kernel boot parameter (e.g., isolcpus=2-31) that tells the scheduler not to place load‑balanced tasks on the listed CPUs. It does not prevent explicit affinity calls such as taskset or sched_setaffinity, which can still bind your strategy thread to the isolated core.

Note: numactl --physcpubind refuses to bind to isolated CPUs because they are omitted from the default mask; use taskset or sched_setaffinity instead.

isolcpus

only affects the scheduler’s automatic load‑balancing; it cannot stop timer interrupts, RCU callbacks, or hardware IRQs.

2.1 Flags

isolcpus=domain,managed_irq,2-31

domain

: default behavior – remove the CPUs from the scheduling domain. managed_irq: keep kernel‑managed IRQs (e.g., NVMe multi‑queue interrupts) away from the isolated CPUs.

2.2 Runtime alternative

Newer kernels recommend the cgroup v2 cpuset controller, which can achieve the same effect without reboot:

# mount cgroup v2
echo "+cpuset" > /sys/fs/cgroup/cgroup.subtree_control
mkdir /sys/fs/cgroup/hft
echo "2-31" > /sys/fs/cgroup/hft/cpuset.cpus
echo "isolated" > /sys/fs/cgroup/hft/cpuset.cpus.partition

For a fixed‑function trading machine, a boot‑time isolcpus configuration is often simpler and easier to audit.

3. nohz_full : Disabling the local timer

Setting nohz_full=2-31 enables “full dynticks”: when a CPU has only one runnable task, the kernel stops sending local timer interrupts, eliminating periodic tick processing on the isolated core. nohz_full=2-31 Constraints:

Exactly one thread must run on each isolated CPU; adding a second runnable task instantly re‑enables the tick.

System‑call entry/exit becomes more expensive because the kernel must read the TSC and perform full memory‑order operations for each transition; hot‑path syscalls, logging and page faults should be avoided or pre‑allocated (e.g., mlockall).

A stable TSC is required ( constant_tsc and nonstop_tsc on x86); otherwise the kernel refuses full dynticks.

At least one non‑isolated CPU must remain; the kernel keeps CPU 0 free for housekeeping tasks.

Even with nohz_full, a residual 1 Hz tick remains for some scheduler statistics; newer kernels offload it to a non‑isolated CPU.

4. rcu_nocbs : Moving RCU callbacks off the isolated core

Typical configuration combines the three parameters:

isolcpus=2-31 nohz_full=2-31 rcu_nocbs=2-31

rcu_nocbs

moves RCU callbacks to the per‑CPU kernel thread rcuo, which runs on non‑isolated CPUs. The kernel automatically adds the CPUs listed in nohz_full to the NOCB set, so the explicit rcu_nocbs flag is redundant but harmless.

5. Writing the parameters to GRUB and verifying

Add the combined line to GRUB_CMDLINE_LINUX_DEFAULT (example for a 32‑core machine, leaving CPUs 0‑1 as non‑isolated):

isolcpus=domain,managed_irq,2-31 nohz_full=2-31 rcu_nocbs=2-31 irqaffinity=0-1

Regenerate the GRUB config and reboot:

# CentOS / RHEL / Amazon Linux
sudo grub2-mkconfig -o /boot/grub2/grub.cfg
# Ubuntu / Debian
# sudo update-grub
sudo reboot

5.1 Confirm parameters were accepted

Check /proc/cmdline for the raw boot line, then verify the kernel‑parsed sets:

$ cat /proc/cmdline
BOOT_IMAGE=... quiet isolcpus=2-31 nohz_full=2-31 rcu_nocbs=2-31
$ cat /sys/devices/system/cpu/isolated
2-31
$ cat /sys/devices/system/cpu/nohz_full
2-31

If the sysfs files are empty, the kernel rejected the parameters (e.g., typo, out‑of‑range CPU number, missing kernel support).

5.2 Verify no other tasks run on isolated CPUs

Use ps or htop to ensure only per‑CPU kernel threads and your strategy thread appear on CPUs ≥ 2:

$ ps -eLo psr,tid,comm --no-headers | awk '$1 >= 2'

Typical leftover threads are migration/N, ksoftirqd/N, cpuhp/N, idle_inject/N —they have fixed affinity and consume no CPU when idle.

5.3 Confirm tick is stopped

Read the LOC column in /proc/interrupts or use perf to count local timer events on an isolated CPU:

$ grep LOC /proc/interrupts
LOC:  4821334  4790012  1204  1198 ...

On a correctly configured core the increase over a 10‑second interval should be single‑digit to low‑double‑digit (residual 1 Hz tick). Thousands of ticks indicate nohz_full is not active, usually because more than one runnable task is present.

5.4 Quantify remaining noise with rtla osnoise

The Real‑Time Linux Analysis suite can run a CPU‑saturating thread on the isolated core and classify the time stolen by various sources:

$ sudo rtla osnoise top -c 3 -d 10s

The output shows overall availability (close to 100 %) and per‑source microsecond consumption for hardware IRQs, soft IRQs, thread switches, etc. If a category exceeds expectations, the corresponding kernel flag is likely missing.

6. Moving hardware IRQs off the isolated core

The boot‑time irqaffinity sets the default mask for IRQs that inherit the default affinity. It does not affect IRQs whose drivers set explicit affinity (e.g., NVMe queues). After boot, you must adjust IRQ affinity manually.

6.1 Stop irqbalance

irqbalance

periodically re‑assigns IRQs, undoing manual changes. Disable it:

sudo systemctl stop irqbalance
sudo systemctl disable irqbalance

6.2 Bind IRQs to non‑isolated CPUs

Iterate over all IRQs and write the desired mask (e.g., 0-1) to /proc/irq/<n>/smp_affinity_list:

for irq in /proc/irq/*/; do
  echo 0-1 > "${irq}smp_affinity_list" 2>/dev/null
done

Writes that fail correspond to kernel‑managed IRQs (NVMe queues). Those are precisely the IRQs that isolcpus=managed_irq is meant to keep away from isolated CPUs.

6.3 Verify effective affinity

Compare the requested mask with the kernel‑effective mask:

for irq in $(ls /proc/irq | grep -E '^[0-9]+$'); do
  name=$(awk -v i="$irq:" '$1==i{print $NF}' /proc/interrupts)
  [ -z "$name" ] && continue
  printf "IRQ %-3s %-26s req=%-8s eff=%-8s
" "$irq" "$name" \
    "$(cat /proc/irq/$irq/smp_affinity_list 2>/dev/null)" \
    "$(cat /proc/irq/$irq/effective_affinity_list 2>/dev/null)"
done

On the example machine, NIC queues end up on CPUs 0‑1, while NVMe queues remain on CPUs 14, 15, 31 as expected.

7. Common pitfalls

Parameters set but no thread bound : isolcpus only frees the core; you must still bind your strategy thread with taskset or sched_setaffinity.

Multiple threads on one isolated core : defeats nohz_full; keep a one‑to‑one thread‑to‑CPU layout.

Binding to an SMT sibling : without disabling hyper‑threading, two logical CPUs share the same execution units, causing contention.

BIOS or kernel upgrades reverting the configuration : verify /sys/devices/system/cpu/isolated after upgrades.

Relying only on /proc/cmdline : the kernel may reject the options; always check the sysfs files and tick counts.

System calls on isolated CPUs : increase latency because nohz_full makes each entry/exit more expensive; mitigate by pre‑allocating memory, locking pages ( mlockall), and off‑loading logging.

Running in virtualized environments : host‑level scheduling and steal time ( vmstat st) introduce noise that cannot be eliminated from the guest.

8. Summary of isolation knobs and verification checklist

isolcpus=domain,...

– removes scheduler‑placed tasks (GRUB). nohz_full=... – removes periodic timer interrupts (GRUB). rcu_nocbs=... – removes local RCU callbacks (already implied by nohz_full) (GRUB). isolcpus=managed_irq,... – removes kernel‑managed IRQs such as NVMe (GRUB). irqaffinity=... – sets default IRQ placement during boot (GRUB).

Stop irqbalance + set smp_affinity_list – removes regular hardware IRQs (runtime). sched_setaffinity / taskset – places your thread on the isolated core (application/runtime).

Verification checklist (run on each isolated CPU):

cat /sys/devices/system/cpu/isolated      # kernel accepted isolation set
cat /sys/devices/system/cpu/nohz_full      # kernel accepted no‑tick set
ps -eLo psr,tid,comm --no-headers | awk '$1 >= 2'  # only kernel threads + your thread remain
sudo perf stat -e irq_vectors:local_timer_entry -C 3 -- sleep 10  # tick count near zero
cat /proc/irq/*/effective_affinity_list   # IRQs land on non‑isolated CPUs
sudo rtla osnoise top -c 3 -d 10s          # residual noise breakdown

Missing any of these steps can leave a spike in the p99.9 latency tail, which may be mistaken for occasional jitter. Thorough per‑item verification is essential for reliable low‑latency deployments.