How Alibaba Achieves Large‑Scale Stateful Container Migration

Background

Alibaba migrated its e‑commerce services (Taobao, Tmall) to containers over 3‑4 years, achieving 100 % container adoption across thousands of nodes. While containers improve resource utilization, many legacy workloads remain stateful, leading to “rich containers” that embed management agents and complicate migration.

Challenges of Stateful Container Migration

Stateful pods occupy physical machines; moving them without disruption is difficult. Kubernetes expects pods to be stateless and uniquely identified, so pod name, UID and IP conflicts block straightforward migration. The execution layer (runC + CRIU) lacks a production‑ready checkpoint/restore mechanism, especially for resources such as host devices, shared file locks, or kernel objects.

Management‑Plane Migration Approach

Alibaba’s solution creates a placeholder pod that mirrors the original pod’s resource specification (CPU, memory, storage, IPs, network interfaces). An OCI‑compatible agent runs on the source and destination nodes and synchronizes the pod state via the CRI. After the new container image is injected into the placeholder, traffic is switched once the old pod reaches a quiet period, and the old pod is retired.

Generate a new pod with identical spec (resources, IP, volume mounts, network cards).

Establish a bidirectional state‑transfer channel between the source and destination agents (runC or Alibaba’s pouch‑container).

Inject the target container image into the placeholder pod through the CRI.

Monitor the source pod for a quiescent window, then cut traffic to the new pod.

Update identifiers (IP, SN) on the new pod, clean up the old pod, and notify the API server of completion.

Key obstacles include handling duplicate pod UIDs, IP address conflicts, and ensuring that controllers such as ReplicationController (RC) or custom IP managers do not automatically recreate the old pod during migration.

RunC + CRIU Migratability

RunC relies on CRIU to checkpoint a process’s memory, registers, file descriptors and network sockets. Memory and register state can be restored accurately; network sockets are restored in “repair mode”, which suppresses actual packet transmission and only synchronizes internal buffers. Migration time is dominated by the amount of memory to checkpoint—large Java processes take longer.

Limitations arise when a process depends on host devices, shared file locks, or kernel resources that CRIU cannot capture, making such workloads unsuitable for live migration.

Emerging Lightweight Runtimes

New runtimes such as Kata, Firecracker, gVisor and Alibaba’s experimental runtimes adopt process‑level virtualization, are often rewritten in Go, and aim for ≤ 5 % performance overhead while providing stronger isolation. Because they have a smaller code base, they tend to have fewer bugs and are easier to audit.

Evaluation criteria include:

Runtime efficiency (CPU, memory, network overhead).

Security isolation (kernel attack surface, syscall filtering).

Code complexity (lines of code, language safety).

Alibaba plans to open‑source a benchmark suite that scores runtimes similarly to automotive safety ratings.

Overall Migration Strategy

The combined approach—resource‑identical placeholder pods, OCI‑agent state synchronization, minimal changes to Kubernetes control loops, and optional use of lightweight runtimes—enables large‑scale, low‑downtime migration of stateful containers in production clusters.