Mixed‑Tenant Architecture: From Exclusive to Shared, Boosting Utilization to 45%

During an annual cost review the finance team highlighted that a company operating three separate clusters—20,000 online servers, 15,000 offline batch servers, and 8,000 AI training nodes—spent a huge portion of its budget on idle resources. Monitoring showed the online cluster averaged only 12% CPU utilization (peaking at 60% during the day, dropping below 5% at night), while the offline cluster averaged 25% (busy at night, idle by day).

Why Exclusive Clusters Waste Resources

Online services are sized for peak QPS. A payment service that normally handles 50 k QPS must provision enough machines for a 300 k QPS promotion, meaning the same machines sit under‑utilized for most of the year. This peak‑oriented sizing creates a large “valley” of wasted capacity, keeping average utilization below 20% even with auto‑scaling, because scaling latency and SLA guarantees prevent aggressive down‑scaling.

Mixed‑Tenant (Co‑location) Architecture

The core insight is that latency‑sensitive online workloads (LS) and best‑effort offline batch jobs (BE) have complementary temporal patterns. By co‑locating them on the same machines, the idle capacity of one can be filled by the other, raising overall utilization.

Time‑dimension staggering: Online services consume ~60% CPU during the day; offline jobs run at night when online load drops to ~5%.

Space‑dimension filling: Even at peak, online services reserve a safety margin (e.g., 40% of CPU) for sudden spikes; offline jobs can use this margin because they are best‑effort.

Google’s Borg paper and Alibaba’s production experience both report utilization jumps from ~10% to >45% after applying mixed‑tenant scheduling.

Isolation Techniques

To prevent offline jobs from harming online latency, resources must be isolated at multiple levels:

CPU isolation: cgroup cpu.quota limits offline CPU; cpuset pins online services to dedicated cores; NUMA awareness keeps memory local.

Memory isolation: cgroup memory.limit_in_bytes caps offline memory, triggering OOM for excess usage.

Disk I/O isolation: blkio (cgroup v1) or io controller (cgroup v2) throttles IOPS and bandwidth.

Network I/O isolation: tc or eBPF rate‑limits offline traffic.

However, software quotas cannot control shared hardware resources such as L3 cache (LLC) and memory bandwidth, leading to the classic “noisy neighbor” problem.

Noisy Neighbor Mitigation

Intel’s Resource Director Technology (RDT) provides:

CAT (Cache Allocation Technology): partitions LLC ways per workload, protecting online cache lines.

MBA (Memory Bandwidth Allocation): caps memory bandwidth for offline jobs, preserving bandwidth for online services.

With CAT and MBA, the hidden interference from shared caches and bandwidth is eliminated.

QoS Tiering and Elastic Throttling

After isolation, a priority system tells the scheduler which workloads to sacrifice when resources are scarce. Kubernetes defines three QoS classes: Guaranteed (online), Burstable , and BestEffort (offline). Kernel‑level enhancements (e.g., Alibaba’s Group Identity) give online cgroups higher scheduling weight.

The decisive mechanism is elastic throttling : when a sudden traffic spike occurs (e.g., a celebrity tweet causing a 5× surge), the system instantly reduces or pauses offline tasks, returning CPU to online services. This decision is driven by real‑time signals such as online latency, PSI (Pressure Stall Information), and CPI (Cycles Per Instruction).

Dynamic Overcommit (超卖)

Online services often request more CPU than they actually use (e.g., 16 cores, but only 4 are needed 95% of the time). Overcommit re‑assigns the unused portion to offline jobs. The ratio is adjusted dynamically: more overcommit during low‑load periods, less when a peak approaches. Open‑source projects like Alibaba’s Koordinator, Tencent’s Caelus, and ByteDance’s Katalyst implement real‑time load profiling to compute safe overcommit levels.

Evolution Stages

The migration from exclusive to unified scheduling proceeds through five levels:

Exclusive clusters: peak‑sized, low utilization, simple but costly.

Online‑only sharing: multiple online services share a pool, modest utilization gain.

Offline mixing: best‑effort batch jobs fill idle cycles, utilization jumps.

Elastic mixing: adds second‑level throttling and dynamic overcommit, pushing utilization to its limit while protecting SLA.

Unified scheduling: a single cloud‑native scheduler (e.g., Google Borg, Alibaba Koordinator) orchestrates both online and offline workloads, handling interference detection and overcommit decisions globally.

The price of higher utilization is a steep increase in system complexity: teams must master cgroups, RDT, interference sensing, and sophisticated scheduling policies. Without these, the risk to online latency is unacceptable.

Conclusion

For workloads at the ten‑million‑QPS scale, the primary rule is that online SLA must never be compromised. The practical solution combines:

Online services running in Guaranteed QoS with CPU pinning and LLC isolation.

Offline jobs in BestEffort mode, designed to be preemptible and resumable.

A unified scheduler that monitors latency, PSI, CPI, and dynamically adjusts overcommit and throttling thresholds.

This architecture squeezes idle capacity to the maximum while keeping latency stable, but the trade‑off between higher utilization and system stability must be carefully balanced.