Unlocking GPU Efficiency: Baidu’s Dual‑Engine Container Virtualization for AI

How to achieve maximum hardware efficiency is a key concern for AI operators and users. Baidu, as a leading AI company, shares its solutions and best practices for GPU container virtualization in complex AI scenarios.

GPU Utilization Challenges

Model training and inference demand exponential growth in compute power, while real‑world usage often suffers from low utilization due to hardware waste. OpenAI data shows compute needs double every 3.4 months; Facebook’s 2021 analysis reports AI GPU utilization below 30% because of faults, scheduling, and resource fragmentation.

Four typical utilization patterns are identified:

Low‑average: peak GPU usage around 10%.

Peak‑valley: daytime peaks, night‑time valleys, average ~20%.

Short‑spike: occasional peaks up to 80% with overall average >30%.

Periodic: batch jobs run for a few minutes every 15 minutes, leaving GPUs idle most of the time.

GPU Virtualization Architecture

Baidu’s “dual‑engine” GPU container virtualization combines a user‑space isolation engine and a kernel‑space isolation engine to meet diverse isolation, performance, and efficiency requirements. The architecture includes a resource‑pooling layer and a unified scheduler (Matrix/k8s) that supports various mixing strategies such as shared, preemptive, time‑slice, and tidal mixing.

User‑Space Isolation Engine

The user‑space engine hooks CUDA APIs, intercepts resource‑related calls, and enforces limits on compute and memory. It is transparent to applications; library replacement is handled automatically by the container engine. Features include memory isolation, compute isolation, encoder isolation, high‑priority preemption, memory over‑commit, and memory pooling.

Kernel‑Space Isolation Engine

The kernel‑space engine implements isolation at the driver and hardware layers, providing fine‑grained memory (1 MB) and compute (1 %) partitioning. It supports Fixed Share, Equal Share, Weight Share, and Burst Weight Share scheduling algorithms, and works with major GPUs (P4, V100, T4, A100/A30) without requiring changes to the user‑space environment.

Mixing Strategies

Based on workload characteristics, Baidu defines several mixing policies:

Shared mixing : multiple low‑utilization tasks share a GPU, achieving >2× utilization.

Preemptive mixing : high‑priority online inference preempts low‑priority batch jobs at kernel granularity, ensuring latency guarantees.

Time‑slice mixing : a global lock controls memory swap in/out, allowing exclusive GPU access for intermittent training jobs and saving up to 80% of resources.

Performance Evaluation

Benchmarks on MLPerf ResNet‑50 Server show that user‑space isolation delivers the lowest tail latency under high load, while kernel‑space provides stronger isolation at a modest performance cost. Memory isolation is static in kernel‑space, whereas user‑space supports over‑commit.

Best Practices and Production Experience

Baidu has applied these techniques in large‑scale AI services, achieving sustained high utilization and stable operation for over two years. The platform, Baidu Bai Ge AI Heterogeneous Computing, is available for both public and private clouds.

Q&A Highlights

Key questions cover GPU resource control mechanisms, support for NPU virtualization, granularity of user‑space isolation (1 % compute, 1 MB memory), compatibility with different CUDA versions, and deployment on private clouds.