Why Today's Cloud‑Native Runtimes Fail AI Agents and How a New Serverless Architecture Can Fix It

Introduction: The AI "Electric Era"

Alibaba Cloud’s Wang Jian likens cloud computing to a digital "super‑grid" that powers the new AI electricity revolution. Large‑language models (LLMs) act as powerful generators, creating countless AI "appliances" (agents, tools, sandboxes) that demand a runtime capable of delivering intelligent services at scale.

Real‑World AI Scenarios on Function Computing

Four representative enterprise use cases illustrate the requirements of modern AI workloads:

Scenario 1 – MCP Server‑hosted AI tools : High‑frequency, short‑lived tool invocations (50‑100 ms) need millisecond‑level startup and zero‑cost idle periods.

Scenario 2 – Interactive content creation agents : Multi‑turn conversations, context memory, and mixed CPU/GPU tasks require persistent session state and low‑latency streaming responses.

Scenario 3 – Personalized AI customer service agents : Event‑driven activation, massive concurrent sessions, and 7×24 availability demand rapid scaling and efficient resource usage.

Scenario 4 – Consumer‑facing AI‑generated content platforms : Massive burst traffic, large file handling, and on‑the‑fly code execution call for ultra‑fast sandbox startup and massive concurrency.

Core Runtime Requirements Derived from the Scenarios

Lightweight, high‑efficiency session management supporting millions of concurrent conversations.

Native support for streaming protocols (SSE, WebSocket) to enable typing‑like output.

Secure, fast‑startup sandboxes with strong isolation and per‑session storage.

Fine‑grained heterogeneous scheduling for CPU‑intensive LLM work and GPU‑intensive diffusion models.

Extreme elasticity: instant scaling from zero to thousands of instances.

Cost‑effective idle handling: "shrink‑to‑zero" while preserving state for rapid wake‑up.

Robust observability and open‑standard integration (OpenTelemetry, JWT, gRPC, etc.).

Why Kubernetes Struggles with AI Workloads

Kubernetes was built for long‑running services. Its central etcd store, strong consistency model, and watch‑based controllers become bottlenecks when billions of short‑lived functions are created and destroyed. Specific challenges include:

Write‑amplification in etcd during massive function creation.

State‑update storms from frequent pod status reports.

Watch overload causing controller latency.

Poor fit for per‑session isolation and rapid scaling.

Additionally, default pod isolation (shared kernel) is insufficient for executing untrusted AI‑generated code, while stronger isolation (MicroVM, gVisor) incurs performance and operational overhead.

Opportunities with Serverless and Function Computing

Serverless offers event‑driven execution, on‑demand resource allocation, and zero‑maintenance operation, aligning with AI agents' bursty, unpredictable traffic patterns. By extending function computing with:

Native session primitives (session‑aware APIs, idle callbacks).

MicroVM‑based sandboxes for strong isolation and millisecond startup.

Heterogeneous resource orchestration (CPU, GPU, XPU) per function.

Dynamic cost models that charge only for active compute, not idle capacity.

These capabilities close the gap between AI workload characteristics and platform capabilities.

Roadmap to an AI‑Native Serverless Runtime

Session‑aware requests : Treat each request as a stateful session, exposing session lifecycle APIs to developers.

Redefined event model : Support rich event sources (message queues, DB changes, IoT) as first‑class triggers for agents.

Secure sandbox revolution : Deploy per‑session MicroVM sandboxes with fast snapshotting and storage isolation.

Heterogeneous orchestration : Schedule CPU‑bound LLM steps and GPU‑bound diffusion tasks to the appropriate hardware pool.

End‑to‑end observability : Integrate OpenTelemetry, JWT, and standard protocols (WebSocket, gRPC, SSE) for full visibility.

Cost model evolution : Introduce idle‑aware billing that charges only for actual compute while preserving long‑lived session state.

Developer experience : Provide "DevPod" environments that mirror cloud runtime locally, enabling seamless debugging and iteration.

Conclusion

To power the next generation of AI agents, the cloud runtime must evolve from a static, resource‑centric OS to a dynamic, session‑centric, event‑driven platform. By embracing serverless principles, secure micro‑VM sandboxes, and fine‑grained heterogeneous scheduling, function computing can become the "super‑grid" that fuels the AI electricity revolution.