How Agentic AI Drives the Evolution from Compute Power to Full‑Stack Infrastructure

As large‑model capabilities expand, the limiting factor shifts from algorithms to the supporting AI infrastructure. Based on a 2026 Agentic AI Super‑Intelligent Systems summit talk, the article systematically describes how Alibaba Cloud's PAI platform builds a full‑stack infrastructure for Agentic AI across three main pillars: compute, inference, and scenario engineering.

Infrastructure Layers

The lowest layer is compute , where PAI manages diverse heterogeneous resources—various CPUs, GPUs (including NVIDIA, Alibaba PPU, and domestic GPUs), high‑bandwidth networks, and storage. Managing a pool of tens of thousands of cards, the platform’s resource pool and elastic scheduling determine the AI system’s upper bound.

Above compute are the two core AI stages: training and inference . Training focuses on GPU utilization (MFU), aiming to raise MFU from the teens to over 50% or higher. Recent advances in LLM serving—such as PD separation, multi‑level KV‑Cache (on‑chip, cross‑machine, SSD), and MTP techniques—boost inference efficiency, forming the foundation for AI production.

The next layer is the enterprise token service (TokenWorks). With high‑throughput token generation, the service guarantees strict SLOs (e.g., 99.9999% latency targets for TTFT and TPOT) and minimizes token cost, which is critical for Agentic AI reliability.

The top layer is scenario‑driven AI engineering , turning tokens into business value across domains such as autonomous driving, embodied intelligence, AI coding, and multimodal exploration.

PAI Unified Scheduling Engine

PAI’s core capability is unified resource scheduling . It handles heterogeneity in compute (different GPU types) and network topology (same machine, rack, cross‑rack, cross‑region) by considering network affinity (e.g., placing Tensor‑Parallel tasks on high‑speed interconnects). Scheduling policies include FIFO, Round‑Robin, and pre‑emptive options, with quota‑based usage modes (fixed quota vs. idle‑time reuse). This engine manages a cluster of over 100,000 cards with >90% effective utilization.

Three‑Tier Resource Management

Purchase stage : PAI introduces an AI resource group concept to centrally manage GPU, CPU, memory, and storage, providing observability and control.

Allocation stage : Quotas isolate team resources, ensuring non‑interference.

Usage stage : Within AI workspaces, resources are bound to specific tasks (training vs. inference), enabling fine‑grained control.

This three‑tier system enables precise compute management, a key reason for PAI’s industry recognition.

PAI‑DLC Large‑Scale Distributed Training

PAI’s training engine (DLC) supports pre‑training, post‑training, MoE, and multimodal models. For MoE training, the Chunk Flow technique converts variable‑length contexts into equal‑length chunks, achieving roughly a 3× boost in resource efficiency. Over the past year, the managed card count grew threefold, MoE MFU exceeded 80%, and monthly training tasks surpassed 40 million.

Agentic Inference Service

Agentic inference introduces four key characteristics:

Long‑context, multi‑round dialogues (tens of thousands of tokens) requiring prefilling and decoding separation.

KV‑Cache as a core asset, with hit rates rising from 60‑70% to >95% in coding scenarios; a 1% increase from 95% to 96% saves ~20% GPU resources.

KV‑Cache‑aware gateways replace traditional round‑robin hashing to preserve cache efficiency and support multiple protocols (Anthropic, Responses, etc.).

SLOs now focus on token‑level metrics (TTFT, TPOT, token cost) rather than raw latency, reflecting the shift to token‑based service granularity.

TokenWorks delivers enterprise‑grade inference with high cache hit rates (90‑95%), operator fusion for various attention mechanisms (e.g., DeepSeek V4, K3), model warm‑up caches, and token management, ensuring cost‑accountable and secure token services.

Scenario‑Centric AI: Physical AI

Physical AI combines simulation of the physical world with LLM/VLM knowledge. PAI provides a full‑stack capability: unified compute management, integration of leading simulation and training frameworks, and a scenario layer offering Notebook Gallery templates, interactive DSW environments, and low‑latency robot simulation. The end‑to‑end pipeline includes data collection (real‑world and simulated), data processing, model training (pre‑training, fine‑tuning, RLHF), and evaluation.

Agentic PAI Interface

Agentic PAI exposes platform capabilities via natural‑language interaction (CLI or chat), covering resource management, training, deployment, operations, and troubleshooting. This enables seamless integration of PAI functions into enterprise AI platforms, benefiting algorithm engineers, deployment engineers, and cost managers alike.

Full‑Modal Data Processing Pipeline

Data remains central to AI. The pipeline spans data governance, resource management, development experience, and multimodal data ingestion. Alibaba Cloud’s big‑data platform evolves from a traditional data warehouse to an AI‑native data platform, allowing AI inference to be expressed as a single SQL statement and supporting heterogeneous resource scheduling across diverse models.

Conclusion

The evolution from raw compute to intelligent agents illustrates a clear trend: AI infrastructure is shifting from sheer compute capacity to sophisticated, token‑aware, scenario‑driven systems that empower Agentic AI across industries. Alibaba Cloud’s big‑data AI platform commits to supporting this transformation.