One Storage Array Powers an Entire Inference Cluster: FN Neo's KV Cache Architecture

As AI shifts from training-centric to inference-centric infrastructure, KV Cache has evolved from a transient buffer into a core data asset with generation cost, reuse value, and lifecycle management needs. Conventional wisdom holds that centralized storage is ill-suited for inference due to elasticity, protocol, and real-time mismatches, yet decades of proven pooled sharing in finance and telecom show it can be upgraded into a shared KV Cache pool.

FN Neo: Centralized All-Flash Storage Designed for AI Inference

1. Pooled Sharing: From Single-Node to Cluster-Level

FN Neo extends traditional NVMe pooling across nodes, turning storage capacity from a "server resource" into a "cluster resource" and KV Cache from "node-private cache" into "inference-cluster shared resource." In a tiered storage architecture, FN Neo becomes the first centralized all-flash tier in the KV Cache hierarchy, allowing multiple compute nodes to share one storage system. It supports both scale-up (adding JBODs without CPUs for low-cost capacity) and scale-out (adding controller enclosures for synchronized performance and capacity growth) up to 1,024 controllers for data-center-scale inference.

2. Native KV Semantics: Minimal Protocol, Sub-Millisecond Latency

Unlike file-system-mediated KV Cache access, FN Neo exposes a native KV protocol with end-to-end latency in the sub-millisecond range. An innovative cache-line-based index locates any KV entry on NVMe within 300 ns, while metadata stays resident in the cache layer for efficient space management.

3. Super Tunnel: Eliminating Hardware and Software Contention

Built on 20+ years of super-tunnel technology, FN Neo removes resource contention at both layers. Hardware contention is eliminated by pinning CPU cores, local memory, network, and drives into isolated fast resource lines. Software contention is removed by bypassing the OS kernel entirely, using lock-free inter-service communication plus XNIO/XDIO zero-copy techniques to minimize data-path overhead.

4. Three-Level Load Balancing: Single Volume, Full Array Performance

Three-tier load balancing ensures a single volume — whether file, block, or native KV — can saturate the entire array, removing the need to create many parallel volumes to achieve peak throughput.

Four Benefits: Fast, Stable, Open, Renewed

Fast: Accelerating KV Cache

Inheriting the high-end FN series performance, a single array delivers 160 GB/s block bandwidth, 70 GB/s NAS bandwidth, and sub-millisecond latency. Verified results show 73% TTFT reduction and 150% token-throughput increase. Centralized all-flash outperforms local NVMe by roughly 2× across latency, throughput, and concurrent-request handling.

Stable: "Storage Faults Without Service Interruption" for Long-Run Inference

Long-context inference is highly sensitive to tail latency (P99, P99.9). FN Neo guarantees stable TTFT, TPOT, and end-to-end latency through four mechanisms: global mirror cache (controller failure without data loss or service interruption), RAID-OC (industry-leading drive-fault tolerance), auto-degraded rebuild (performance non-degradation during rebuild), and sub-health/slow-drive auto-isolation (preventing single-drive drag on the whole cluster).

Open: Preserving Existing AI Inference Ecosystems

FN Neo offers "triple openness": framework compatibility with vLLM, SGLang, LMCache, HiCache, Mooncake; networking support for RoCE, IB, domestic 400G IB, FC, IP SAN — simultaneously serving "NVIDIA GPU + Mellanox IB" and "Domestic GPU + Domestic IB"; and OS compatibility across mainstream Linux distributions.

Renewed: From "Store It" to "Store Fast, Stable, Economical"

Cluster-level KV Cache pooling upgrades storage from passive capacity to active infrastructure that defines compute efficiency.

Measured Data: From Lab to Production

1. Single-Node Test

An 8-GPU node running DeepSeek-R1 with FN Neo KV offload achieved 7–12× context-length performance gains across varying context sizes.

2. Multi-Node Test

Two 8-GPU nodes running Qwen-2.5 showed 6–10× speedup with offload enabled versus disabled.

3. Real-World AI Coding Scenario

Continuous context growth and multi-turn iterations make AI coding the most storage-dependent workload. Measured first-token latency acceleration exceeded 6×.

Ecosystem Compatibility & Zero-Intrusion Integration

1. Framework Compatibility

Native KV protocol supports Sugon's newly released ParaCache manager and plugs directly into LMCache, HiCache, or as Mooncake's KV backend.

2. Networking & OS Compatibility

Same multi-protocol and multi-OS support as above.

3. Zero-Intrusion Deployment

A lightweight user-space client component connects compute nodes to the storage backend without kernel changes, acting as middleware between the model and storage.

Conclusion: Storage Moves from Passive Support to Active Compute-Efficiency Participant

Through cluster-level KV Cache pooling, native KV semantics, super-tunnel contention-free paths, and open ecosystems, FN Neo redefines AI inference infrastructure standards — making storage "fast, stable, and economical" rather than merely "capacious."