TensorCast Cuts First‑Token Latency by Up to 93.2% with Unified Programmable Tensor Management

Recent large‑model inference optimizations have focused on making GPUs compute weights, activations, and KV cache more efficiently, but growing model sizes, longer contexts, and multi‑turn Agent applications create a new challenge: how to flexibly and efficiently manage massive tensor state that flows across system components.

Current infrastructures treat each tensor‑related task—weight loading, KV‑cache storage, checkpointing—as separate, tightly coupled systems. Although each system improves performance, they duplicate effort and lack a unified abstraction for tensor management.

To address this gap, Peking University, Jietie Xingchen, and Beijing University of Posts and Telecommunications jointly propose TensorCast , a Tensor‑as‑a‑Service (TaaS) abstraction that decouples tensor lifecycle from specific execution, storage, or network layers. By providing a unified, programmable management layer, TensorCast can reduce the first‑token latency (TTFT) of high‑concurrency multi‑turn Agent workloads by up to 93.2% while matching dedicated system performance.

The system is built on two core ideas: (1) tensors become first‑class system objects with independent identity and lifecycle, allowing the platform to manage weights, KV cache, and other intermediate states without the developer needing to know their physical location; (2) the tensor lifecycle is programmable through primitives—Identify, Place, Move, Transform, Materialize—enabling developers to compose custom optimization strategies. For example, a KV‑cache migration across inference instances can be expressed as a simple sequence of export, transfer, and restore operations.

TensorCast adopts a distributed architecture that separates control and data planes. A Global Store maintains cluster‑wide metadata (node status, tensor locations, scheduling info) without handling data transfer, while multiple Worker nodes manage local storage, memory, and GPU resources and execute lifecycle operations. Workers communicate via RDMA, zero‑copy, and pipelined transfers to achieve high‑performance peer‑to‑peer data movement and scale with cluster size.

Experimental validation integrates TensorCast into the popular inference frameworks vLLM and SGLang. Results show that model startup time in elastic scaling scenarios improves by up to 228.6× compared with common distributed file systems, KV‑cache performance matches the specialized Mooncake system, and a programmable scheduler for multi‑turn Agent workloads lowers median TTFT by 93.2% . These findings confirm that a generic tensor‑management layer can deliver both performance and programmability.

In summary, TensorCast proposes a new tensor‑management paradigm for large‑model infrastructure, decoupling critical state from computation and enabling programmable, cross‑component optimizations. As LLM applications become more dynamic—supporting agents, continual learning, and multi‑turn interactions—such unified lifecycle management is poised to become a foundational capability for next‑generation AI systems.