Boosting Test Environment Stability: Automated Container Replacement & Buffer Pools

Pain Points

Frequent container allocation failures cause development and testing to stall, leading to hundreds of lost hours each day. After a half‑year rollout of Pouch, daily container requests increased tenfold, but low success rates created a major bottleneck.

Test‑environment hosts are outdated and over‑committed, making failures common.

When a host fails, its containers are not automatically migrated, causing repeated deployment failures.

The scheduler marks faulty hosts as unschedulable, but containers remain, reducing usable resources.

Resource pools have different priorities and do not share machines, limiting capacity visibility and causing scheduling failures.

Insufficient capacity alerts and lack of optimization for test‑specific workloads further lower success rates.

Goal

The target is a container allocation success rate of 99.9% .

Solution

Data collection and analysis were the foundation of the improvement plan. The team gathered end‑to‑end metrics from Normandy (foundation platform), Huangfeng (resource request), Zeus (second‑level scheduler), and Sigma (global scheduler) to monitor success rates and failure cases.

Key components of the solution include:

Automatic container replacement : When a host is detected as faulty (e.g., disk full, hardware error), the system automatically provisions a new container on a healthy host and decommissions the old one.

Buffer pool : A pre‑warmed pool of containers sits near the user‑facing side of the pipeline. If a direct request fails, a buffer container is allocated, and the pool asynchronously replenishes the used slot, shielding users from underlying failures.

Resource‑pool rationalization : Resources are re‑allocated based on historical demand, peak usage, and priority (ordinary users vs. system users) to improve overall capacity utilization.

Dynamic switches in the buffer pool allow selective disabling of features (e.g., hostname changes) to keep allocation latency under one second.

Health checks before delivering buffer containers ensure only usable containers reach users, and periodic cleanup removes dirty containers.

Conclusion

After implementing the above measures, the container success rate rose sharply, with the buffer pool reducing rate fluctuations dramatically. The two‑month trend shows a stable success rate above 99.9% except for isolated incidents caused by buffer‑pool bugs, which were quickly fixed.

Future Work

Remaining challenges include automating resource‑pool capacity adjustments, dynamically scaling the buffer pool to cover low‑frequency images, and further improving overall resource utilization and container provisioning latency.