How Agents and Native Liquid Cooling Redefine AI Compute Architecture with Millisecond Latency

1. Agents Consume More Than Just GPUs

At the 2026 Open Compute Technology Summit, Inspur’s VP Zhao Shuai argued that AI workloads are moving from isolated model calls to large numbers of agents operating 24/7. In a 33‑agent system, only two inference steps run on GPUs while six steps—branch prediction, information retrieval, logical decision, task orchestration, etc.—run on CPUs. Zhao emphasized that "GPU sets the model’s intelligence ceiling, but multi‑agent engineering raises the practical output quality," meaning both model capability and agent engineering are essential.

This shift changes the CPU‑GPU power ratio: traditional AI centers stacked GPUs, but with agents the demand for CPU resources skyrockets, prompting a focus on how to boost CPU capacity within server racks.

2. Cabinets Near Megawatt Levels: Evolving Liquid Cooling

When agents become permanent 24/7 workers, the physical platform must evolve. Inspur unveiled a "native liquid‑cooled" full‑rack server that fits four CPUs into a 0.5U space, achieving 384 CPUs per rack and supporting over 40,000 concurrent agents. Traditional air‑cooled racks top out at 40‑50 kW; with this density, a single rack’s power draw reaches 3 000 kW, rendering conventional 380 V copper cabling impractical. Inspur’s solution uses 800 V DC busbars and a "power‑separation" design that isolates power modules and provides 100 % liquid cooling, enabling zero‑downtime maintenance.

The article contrasts this native liquid cooling with the common "add‑on" approach where liquid‑cool plates are attached to air‑cooled servers, noting that the latter still retains fans, ducts, and complex piping, limiting density, energy efficiency, and operational simplicity.

Key design principle: compute and cooling are co‑designed, eliminating airflow ducts and allowing double‑sided access, three‑dimensional placement of CPUs, GPUs, memory, SSDs, and NICs, and using double‑sided cold plates with optimized flow channels to match the rapid increase in chip power.

Standardized modular building blocks (compute, intelligence, and switch modules) create a reusable, extensible architecture for AI infrastructure of varying scales.

3. Multi‑Modal Fusion and the 4.77 ms Token Race

Inspur’s previous "Meta‑Brain SD200" supernode could host multiple trillion‑parameter models. The new "multi‑modal fusion" approach sends a single API call to several candidate models, then a reviewer model aggregates results into a more reliable answer.

Benchmark results show fused models outperform single models: 53.9 % higher scores in deep‑research tests, 97.2 % on AIME 2026 math reasoning, and 90.8 % on GPQA Diamond. This requires infrastructure with terabyte‑scale unified memory, millisecond‑level latency, and high‑bandwidth interconnects.

The optimized SD200 reduces Kimi K2.6 token generation from 8.9 ms to 4.77 ms, becoming the first domestic supernode to break the 5 ms barrier. The enterprise‑grade SD200 offers 16‑card interconnect, terabyte‑level VRAM, and reduces first‑token latency (TTFT) by over 40 % for trillion‑parameter models, lowering deployment thresholds.

4. The Dawn of Collective Intelligence: AI’s Industrial Era

Zhao describes three layers of "collective intelligence":

Clear compute division: GPU supernodes handle high‑throughput token generation, while CPU agent hosts manage coordination, scheduling, and sandbox execution.

Model specialization: tasks are routed to appropriate models—OCR to small models, complex code generation to large models, long‑context reasoning to fused models—optimizing token cost.

System‑level collaboration surpasses single‑model capability, turning AI into a cooperative system rather than a solitary smart model.

The transition from air‑cooled to native liquid‑cooled hardware, from single‑model inference to multi‑modal fusion, and from one‑off calls to agent clusters marks a watershed: AI is moving from a model‑centric race to an industrial‑scale, system‑level collaborative era, built for the sustained operation of thousands of agents rather than peak performance of a single chip.