RED OHC Open-Sourced: New Java Off-Heap Cache SOTA, 3.38x Multi-Threaded Throughput

Ten Million Recalls Per Second: What It Means for Cache

In Xiaohongshu's search and advertising pipeline, each request recalls candidate notes from a hundred-million-scale corpus before ranking. The entire pipeline completes tens of millions of note recalls per second. Note content and feature data are mostly read from cache. Each business uses independent cache instances; within a single instance, multiple read and write threads concurrently access the same data.

Because cache sits on the request critical path, it faces two direct requirements:

Latency. Cache queries are on the critical path; single-access budget is tight. Cache P99 affects pipeline P99; thread queuing, GC pauses, or lock contention can propagate to request latency.

Result freshness and consistency. Returned data participates in note recall. Stale data, reading mixed old/new content, or entries accidentally dropped during writes can affect recall and ad delivery. Therefore, TTL expiration semantics, update atomicity, and capacity accounting must be explicit.

At tens of millions of recalls, these requirements translate into concrete engineering constraints:

Read-heavy and highly concurrent. Massive requests continuously hit cache. Fixed cost per lookup is amplified; read path must be short and avoid shared locks causing thread queuing.

Reduce GC impact on request latency. Tens of GBs in JVM heap increases management and collection pressure. Off-heap storage reduces this pressure but does not eliminate GC entirely.

Continuous writes with eviction. New notes enter cache, existing content updates, while capacity is limited; eviction occurs continuously. Eviction policy affects hit rate, which in turn affects back-source volume.

Controlled memory footprint. Hundred-million entries and tens of GBs per machine require controlling per-entry overhead; freed space must be returned to the OS.

Lower fixed operation costs. Hashing, locks, memory barriers, and object allocations can appear on hot paths; their overhead must be weighed against target load.

Thus, the pipeline needs a cache engine supporting off-heap storage, low-overhead concurrent reads, precise TTL, asynchronous eviction, and physical memory reclamation.

Cassandra OHC: Right Starting Point, Unscalable Architecture

For the past two to three years, Xiaohongshu used the open-source Cassandra OHC (Off-Heap Cache) in production. It is the off-heap cache implementation from the Cassandra ecosystem and served as their Java off-heap cache baseline, running stably for over two years.

Under search/advertising access scale and concurrency patterns, load testing revealed four limitations:

Shared lock contention on read path. OHC's read path accesses a shared lock structure within the same cache instance. In this benchmark, single-thread pure read achieved 2.92M ops/s, while 32-thread pure read reached 39.87M ops/s — only 13.7x scaling, far below linear. Multiple reader threads contend at the shared synchronization point, limiting read throughput scaling.

Heap allocations on read path. Under current test configuration, OHC's get allocates ~32 bytes per operation on the Java heap. At 32 threads, read-path heap allocation rate approaches 1.4 GB/s, triggering over 2 Young GCs per second on average. Off-heap storage itself does not eliminate GC impact; allocations on the read path still cause collection pressure and disturb tail latency.

Synchronous eviction in caller thread. When capacity is exceeded, OHC performs LRU eviction in the write request thread. The write thread must handle eviction in addition to updating entries; concurrent writes introduce additional contention.

Underlying memory not returned promptly. After entries are released, underlying memory is not returned to the OS in step with cache capacity changes; process RSS drop is limited. Scaling down often requires process restart.

These limitations stem from coupling between read/write operations and physical memory maintenance. Updating an entry forces the business thread to also handle eviction and partial memory maintenance. In October last year, the team began rewriting the engine, naming it RED OHC.

RED OHC Architecture: Separating Logical and Physical Planes

RED OHC splits each cache instance into a logical plane and a physical plane. Business threads handle queries, logical updates, and new value publication; old value reclamation, eviction maintenance, index reorganization, and free-page return are delegated to a background daemon actor per instance. New value allocation still occurs on the write path but uses per-thread arenas to reduce shared contention.

The logical plane records indexes, key-value mappings, entry logical states, and value pointers. When get or put returns, the logical result of that operation must already be effective. The physical plane manages native memory pages and slots, old value release, and free-page return. Maintenance work not required to complete within the current request is deferred.

Each cache instance has three main roles: reader (business read threads) and writer (business write threads) handle their respective requests; the background daemon thread, called actor, sequentially executes most cross-entry physical maintenance for that instance.

Read: Completely Lock-Free

RED OHC's steady-state get acquires no mutex; multiple reader threads can lookup concurrently without contending for a single lock on the read path. In 32-thread pure-read testing, RED OHC throughput reached 3.38x the OHC baseline. This is the whole-engine result; lock-free reading is one factor.

Lock-free reading must solve when to release old memory. If old memory is freed immediately after an entry update or delete, threads still reading it may access invalid addresses. Therefore, RED OHC defers old memory release until related read operations finish, then the background actor reclaims it.

This can be understood via EBR (Epoch-Based Reclamation): the reclaimer confirms that readers potentially accessing old data have exited before freeing the corresponding memory. RED OHC adopts the same safe-reclamation idea but uses a custom mechanism, not a direct copy of classic EBR implementations.

The read path minimizes extra work and heap allocations. Steady-state read heap allocation approaches zero; TTL is checked at read time, and expired entries are not returned.

Write: Publish-Then-Commit, Physical Reclamation Later

put

allocates space for the new value in the writer thread's private arena, using 64 KB to 2 MB contiguous pages pooled by size class. It then writes data, atomically publishes the pointer, and dispatches a change event. After dispatching, the write thread does not wait for background maintenance; upon return, the logical update is already effective.

Physical release of the old value is handled by the actor. The writer places the old address into its own retirement channel; after relevant readers exit their protection zones, the actor releases pages in batches. Entries carry a version generation; replacement takes effect atomically at the commit point. Readers see either a valid old value or the new value, never a half-updated state.

Actor: Physical Plane Daemon

Cross-entry work — eviction decisions, TTL maintenance, page reclamation, index reorganization — is processed sequentially by the actor in a single thread. This serializes the background maintenance path while business read/write threads continue handling requests concurrently. The actor does not need to synchronize shared state across multiple maintenance threads, and these maintenance tasks do not intrude into request threads.

The writer dispatches change events to the actor and continues processing requests. The actor typically merges events in 200-microsecond time windows; when backlog grows, it switches to per-event response. Batching reduces scheduling overhead at low load; per-event response helps the actor keep up under high load.

Free pages are also returned to the OS by the actor. In internal stress tests, after the test ended, cache RSS dropped from 8.5 GB to 51 MB within 10 seconds, without process restart and without relying on a specific memory allocator.

Hardware-Friendly: Cache-Line Design and Assembly-Level Analysis

Critical engine state is aligned to 64-byte cache lines. Entry metadata is packed in a native prefix; each reader thread's registration slot occupies a dedicated cache line, reducing cross-thread interference. The writer's allocation cursor within a page and the reclaimer's free bitmap are placed separately. The writer reserves a batch of 64 slots via a single CAS, then hands them out from a thread-local mask. This layout reduces multiple cores repeatedly writing to the same cache line, lowering coherence traffic and cache-line ping-ponging.

The team used AI-assisted analysis of JIT-generated machine code, manually verifying hot-path inlining, memory-ordering barriers, and instruction counts. The focus was confirming which instructions a single get actually executes and identifying fixed overheads not obvious from source code.

Data Structures and Algorithms: No JDK Collections, All O(1) on Hot Path

For concurrent data structures, the writer-to-actor change channel uses a lock-free MPSC queue; multi-producer page-pool acquisition uses an xadd variant. These implementations come from JCTools and are optimized for producer/consumer roles with cache-line isolation. For RED OHC, queue selection depends on actual overhead under target concurrency patterns, not just API familiarity.

TTL maintenance uses a hierarchical timing wheel similar to Kafka's. Entries are placed into different levels by expiration time; time spans expand per level, covering up to months. Routine expiration handling is O(1). Bitmaps skip empty time slices; far-future entries fall back to a min-heap. Thus, regular TTL maintenance does not require scanning the entire cache.

Index probing on the hot path, unlinking known eviction nodes, and timing-wheel expiration handling are all designed O(1); memory reclamation merges by page in batches. Compared to implementations requiring full-table scans or linked-list traversals, these operations avoid linear work that grows with entry count.

Beyond Speed: Other Engine Designs

Beyond throughput, real-world use involves eviction policies, API compatibility, TTL testing, and memory observability. Several related designs:

Eviction policies covering different access patterns. Default is S3-FIFO, a 2023 FIFO-family algorithm using two frequency counters and a ghost queue to identify entry reuse; unlinking is O(1). LRU is retained as a compatibility policy. For search/ad recall load, a virtual Skip zone similar to S4-FIFO is added on top of S3-FIFO. Single-access entries enter a small queue and leave FIFO without updating frequency; this reuses the existing 3-bit policy state without extra fields. The ghost queue uses an off-heap open-addressing table with incremental rehash on expansion, avoiding one-time full-table migration. The engine also provides W-TinyLFU with hill-climbing adaptive parameter tuning. Policy computation runs single-threaded in the actor; read threads sample asynchronously at 1/16 probability, avoiding shared-list rearrangement on every hit.

API follows familiar Java engineer calling conventions. Operation interfaces adopt a JDK Map-like style, providing get, put, putIfAbsent, replace, and remove; the builder follows Cassandra OHC's OHCacheBuilder. This reduces re-learning during migration.

Supports direct off-heap data reads. Deserializers can read off-heap read-only views; advanced callers can consume native memory directly via ValueView or DirectValueConsumer, skipping deserialization and extra copies.

Reduces hash collision risk. Admission protection references Caffeine's Hash-DoS handling, lowering risk of malicious collision keys continuously entering the cache.

Eviction callbacks provide reasons. EvictionListener and RemovalCause distinguish eviction, expiration, and explicit deletion, enabling downstream pre-warming or auditing.

Memory usage auditable. The engine accounts capacity by bytes and provides audit counters for each page pool, facilitating inspection of memory distribution and reclamation.

Benchmark

Baseline: Cassandra OHC and current RED OHC version. Cassandra OHC matches the production-serving version and has been bytecode-compared category by category.

Benchmarks ran on AMD EPYC 7T83 (Zen 3) with JDK 11, 1 GB heap, 1 GB native memory. JMH throughput mode, key length 16 bytes, value length 64 bytes, working set 24K entries, uniform random access. RED OHC used default S3-FIFO; Cassandra OHC used default LRU. Five test rounds. Throughput unit: 10K ops/s. The 5-round ratio range shows min-to-max observed per round, not confidence intervals.

JMH GC profiler ran 32-thread pure read for 3 rounds, 3 seconds each. In that measurement window, Cassandra OHC's get averaged ~32 B/op heap allocation with 22 GCs; RED OHC averaged ~0.001 B/op with 0 GCs. This data describes that profiling window and does not imply other loads or long runs never trigger GC.

Under this hardware and test scope, RED OHC throughput exceeded Cassandra OHC baseline in all six scenarios: single-thread read, write, read-write 9:1 at 1.13x, 1.26x, 1.25x respectively; 32-thread read, write, read-write 9:1 at 3.38x, 1.30x, 2.60x respectively, with the largest gap in 32-thread pure read. For the given baseline and load, RED OHC refreshes this set of Java off-heap cache benchmark results. Microbenchmark conclusions are limited to the listed hardware, dataset, and configuration; they cannot be directly extrapolated to full-chain production QPS.

Closing Notes

From adopting Cassandra OHC to initiating in-house development, RED OHC's design has consistently targeted one goal: let request threads handle only the current read/write, offloading eviction, TTL maintenance, and memory reclamation to the background. Lock-free read paths, single-threaded actor for cross-entry maintenance, and mechanisms like S3-FIFO, hierarchical timing wheel, and page-based reclamation form the maintenance substrate. In the new benchmark round, RED OHC outperformed Cassandra OHC baseline in all six scenarios: 32-thread pure read 3.38x, read-write 9:1 2.60x, single-thread read-write mix 1.25x.

RED OHC has completed canary validation and is running in production on Xiaohongshu's core search/recommendation recall pipeline. Benchmarks compare controlled-load performance; production canary observes behavior under real traffic and engineering constraints.