Eliminating Tail Latency in High‑Frequency Trading: Memory, NUMA, and Cache Strategies

1) Clarify the basics: UMA vs NUMA are latency models

UMA (Uniform Memory Access)

On UMA machines any CPU core accesses DRAM with roughly equal latency; cache hit/miss still matters, but there is no distinction between local and remote memory.

NUMA (Non‑Uniform Memory Access)

NUMA machines split CPUs and memory into multiple NUMA nodes (e.g., node0, node1 on dual‑socket servers). Each node has its own memory controller and DRAM channels.

Core accesses its local node’s DRAM → lower latency, higher bandwidth, less variability.

Core accesses a remote node’s DRAM → must cross the socket interconnect (e.g., Intel UPI, AMD Infinity Fabric), usually slower and more jittery under contention.

On NUMA, the critical path fears not only “accessing DRAM” but “accessing remote DRAM”.

2) Why NUMA creates tail latency: from average slowdown to occasional spikes

Remote memory accesses often degrade the tail of the latency distribution because:

Interconnect and remote memory controller queuing: many cores hitting remote memory simultaneously cause bursts of delay.

Cross‑socket cache‑coherence traffic: frequent writes to shared data cause cache‑line migration; crossing NUMA nodes is costlier and less stable.

Scheduler‑induced thread‑to‑data mismatch: a thread may be migrated to another node while its memory pages remain on the original node, turning the critical path into a remote access.

In trading systems this manifests as:

Overall average latency looks fine, but P99/P999 suddenly spikes.

Same binary on the same machine shows different tail behavior after different deployments or restarts due to random thread‑memory placement.

Unchanged business code produces a reshaped latency distribution after a re‑ordering strategy.

3) Core goal of NUMA optimization: data near compute

NUMA tuning aims to:

Pin critical threads to a specific NUMA node’s cores.

Allocate frequently accessed data on the same node.

Avoid cross‑node shared‑write hotspots, reducing remote coherence traffic and cache‑line migration.

This is analogous to “colocation” inside a machine: keep data on the same node.

4) Effective NUMA organization for trading systems: application‑level sharding/partitioning

HFT naturally fits “per‑flow partitioning”, which aligns well with NUMA.

4.1 Partitioning criteria

By instrument/contract: each partition maintains its own market state, order book, risk counters, pricing parameters.

By account/strategy: each partition handles a set of strategies or accounts, reducing write contention on shared risk state.

By session/connection: each trading session or market data channel is bound to a specific thread/core.

4.2 Physical placement of partitions

Partition threads run on the cores of a chosen node (e.g., node0).

Objects such as pools, queues, hash tables, and order‑book structures are allocated on that node’s memory.

Inter‑partition communication uses message passing (one‑way queues) instead of shared large locks.

4.3 Routing stability

If events for the same instrument or account bounce between nodes, data either becomes a shared‑write hotspot or incurs cross‑node movement. Therefore the mapping “event → partition → thread/core → NUMA node” must remain stable.

5) Cache hierarchy: a deeper placement problem (working set and jitter)

NUMA decides which DRAM to access; cache decides whether DRAM can be avoided.

5.1 Working set in the LLC

Critical‑path data should stay hot and small enough to reside in the last‑level cache (LLC), e.g., hot price levels in the order book, key parameters, latest market snapshots, and routing tables. If these items are evicted from LLC, latency distribution widens and tail latency rises.

5.2 Avoiding cache‑line jitter

In trading systems the dominant cache issue is write‑induced line migration:

Multiple threads updating the same structure cause cache lines to bounce between cores.

Two unrelated counters sharing a cache line interfere (false sharing).

Cross‑node shared writes are especially harmful because coherence traffic traverses the interconnect, making tail latency hard to control.

Engineering practice prefers:

Per‑thread/per‑partition private state.

Shared state designed for “read‑many, write‑few”, with writes batched or handled by a single writer.

Counters aggregated per‑thread or per‑core rather than via global atomics.

6) Common NUMA anti‑patterns in HFT

Threads not bound to cores, allowing scheduler migration, which creates remote accesses and tail jitter.

Sharing a global object with frequent writes, turning the system into coherence‑traffic‑driven with costly cache‑line bouncing.

Uncontrolled cross‑node memory allocation (default allocator, containers, multi‑process), leading to idle CPUs but spiky latency.

Building parallelism on excessive cross‑partition calls, defeating the locality benefits of partitioning.

7) NUMA design principles for HFT

Fixed mapping: event flow → partition → thread → core → NUMA node.

Localize state: allocate partition data structures on the same node.

Single‑writer preference: give one thread ownership of hot write‑paths; communicate across partitions via message queues.

Control working set: keep hot path structures small and compact to reduce pointer chasing and random accesses.

Minimize cross‑node sharing, especially frequent atomic ops and lock contention.

Use tail latency as the metric: aim to straighten the latency distribution and shorten the tail rather than optimizing the mean.