CubeSandbox v0.7.0: Cross-Machine Sandbox Migration & Multi-Version Runtime Support

Cross-Machine Pause/Resume via S3 Backend

Previous pause/resume and snapshot-based creation were bound to a single host machine. v0.7.0 uses an S3 backend (default MinIO, user-configurable) to persist sandbox memory and filesystem state to shared object storage. This enables pausing on node A and resuming on node B, or launching new sandboxes from the same snapshot on any synchronized node. The feature is in preview.

Concrete example: a compute node scheduled for maintenance can have its paused sandboxes migrated to other nodes instead of waiting for local resume or discarding them. In agent-training workloads requiring massive parallelism and frequent pause/resume, the scheduler can place sandboxes on the least-loaded nodes based on real-time load.

Component Multi-Version Coexistence

Cross-machine flow introduces a side effect: templates and snapshots built at different times or on different nodes may depend on different runtime component versions. Upgrading components previously invalidated old templates and snapshots, forcing rebuilds. v0.7.0 lets compute nodes retain historical component versions, so templates and snapshots remain valid across upgrades, and existing pause/resume operations are not interrupted.

Before this, a production cluster upgrading to fix bugs might need to invalidate all existing templates and snapshots, halting business to re-bake them. This change decouples upgrades from asset invalidation, a critical shift for long-lived, continuously upgraded clusters.

Network Subsystem Refactor: Merging NetworkAgent into Cubelet

Network creation has been a critical-path bottleneck in sandbox cold starts, especially during batch launches. v0.7.0 merges the formerly independent NetworkAgent into Cubelet, reducing RPC calls in the sandbox creation flow. It also optimizes the eBPF policy distribution path, significantly lowering creation latency for sandboxes with network rules, and improves Tap device allocation stability under high concurrency.

When a single request launches hundreds of sandboxes, each sandbox saves two RPC calls and one RCU wait; these savings scale proportionally with batch size, reducing cold-start time. Post TAP lifecycle refactor, network anomalies during node reboot or sandbox resume occur less frequently.

Control Plane and Operations Architecture Separation

v0.7.0 moves node-management and other operational capabilities from CubeMaster to a new component, CubeOps. CubeOps now handles full node management, defaults to two replicas, and replaces cubemastercli node subcommands with cubeopscli. The Web UI adds node isolation/unsealing and operation-record viewing.

Example: when ops detects a disk anomaly on a node, they can isolate it via the Web UI — existing sandboxes keep running while new ones are scheduled elsewhere. This clarifies responsibility boundaries: scheduling stays with CubeMaster, operations with CubeOps.

Additional Enhancements

Three-Language SDK Parity

Volume CRUD and volumeMounts support, previously Python-only, are now implemented in Go and Node SDKs. Template alias coverage is completed across build-time aliases and existing template alias management for Go, Node, and Python. Python SDK adds distribution_scope to explicitly target nodes or regions; Node SDK supports NEVER_TIMEOUT at runtime; Go SDK introduces Files.ForUser for per-user isolated file views.

Storage Block Device and Template Source Extensions

New SPDK-based CubeS3lvol COW provides a remote copy-on-write block device backed by S3 object storage: local write allocation with async flush to S3, exported to host via NVMe-oF/TCP loopback. Supports snapshot/clone and cross-node import/export, with crash consistency guaranteed by local WAL + journal. v0.7.0 bundles MinIO as default S3 volume backend for out-of-the-box use. Private HTTP image registries can now serve directly as template sources without separate HTTPS setup.

Dynamic Network Policies and Configurable Forwarding

Sandboxes support runtime network policy updates without rebuild. CubeEgress L7 forwarding rules accept custom ports; CubeProxy allows custom management ports and adds plaintext gRPC ingress; CubeVS introduces same-subnet MAC learning to avoid non-hairpin communication blackholes. passfd switches to bare-pipe vsock direct connection, improving business-process I/O efficiency. Kernel and guest image artifacts are now published independently, pulled from fixed kernel-release-* / guest-image-* releases instead of recompiling each version.

High-Frequency Bug Fixes

v0.7.0 addresses production-recurring issues: pause/resume state inconsistencies, zombie processes and deletion anomalies (PRs #978, #985, #1137, #1274), snapshot performance (PRs #1300, #1504), template cache data races during concurrent sandbox creation (PR #1366), and TAP device recovery anomalies on node reboot (PRs #930, #987, #1207).

Roadmap

Post cross-machine capability, CubeSandbox will pursue cloud-native high availability: sandbox anomaly recovery (auto-detect and recover VM crash, shim hang, network partition with configurable strategies — restart, snapshot rollback, or error reporting), and enhanced scheduling/ops (resource-aware scheduling, affinity/anti-affinity, priority classes, online resource rebalancing, and node drain with sandbox migration). Near-term iterations (v0.7.1 or v0.8.0) will deliver control-path full HA (template management split from CubeMaster, CLM multi-replica deployment) and cross-node sandbox recovery on node failure.

Full changelog:

https://github.com/TencentCloud/CubeSandbox/blob/master/docs/zh/changelog/v0.7.0.md

. Repository: https://github.com/TencentCloud/CubeSandbox.