Janus: Dual‑Timescale Scheduling for Production‑Scale Multi‑LLM Serving

Introduction

Janus is a Service–Engine collaborative system for production‑scale multi‑large‑model (LLM) serving, part of the Oxygen xLLM AI infrastructure. It targets three intertwined challenges of large‑model platforms: unpredictable burst traffic, power‑law distributed application hotness, and heterogeneous yet complementary resource requirements.

Motivation

Early LLM inference systems optimized single‑model throughput and latency, but production Model‑as‑a‑Service platforms must simultaneously serve hundreds of models ranging from 0.6 B to 671 B parameters with diverse request rates, input/output lengths, and TTFT/TPOT SLOs. Analysis of production traces reveals (1) most applications have stable traffic while a few experience 5–10× spikes within seconds, (2) the top 12 % of applications generate 80 % of requests (power‑law distribution), and (3) models differ in HBM capacity, compute, and bandwidth needs, offering co‑location opportunities.

System Overview

Janus consists of a centralized Service layer and a distributed Engine layer . The Service layer performs global performance modeling, model placement, elastic scaling, and request dispatch. The Engine layer manages per‑instance memory, compute resources, and model lifecycles. Runtime telemetry forms a closed loop: Engine reports load and performance; Service adjusts decisions on the appropriate time scale.

Dual‑Timescale Scheduling

Two scheduling paths operate concurrently:

Slow path (30 s) : Executes multidimensional vector bin‑packing to co‑locate complementary models in the Steady Pool , improving long‑tail resource utilization.

Fast path (0.5 s) : Uses an elasticity Gaussian‑process (Elasticity GP) to predict SLO achievement and quickly adjust replica counts in the Elastic Pool for head‑heavy, bursty applications.

This separation avoids using a single scheduling interval for both long‑term efficiency and short‑term elasticity.

Performance Oracle

Janus maintains two Gaussian‑process models:

Capacity GP maps workload features (request rate, input/output length, etc.) to a three‑dimensional resource vector (HBM, compute, bandwidth) for the slow‑path placement.

Elasticity GP predicts SLO fulfillment for a given replica count, guiding fast‑path scaling decisions.

Both models work well with sparse observations, providing uncertainty estimates useful for newly added models.

Model Scheduler

The slow path treats model placement as a multidimensional vector‑packing problem; an incremental algorithm reduces the overhead of frequent global solves. The fast path consults Elasticity GP; when current capacity cannot meet load, additional replicas are launched in the Elastic Pool, and excess replicas are reclaimed when load drops.

Request Scheduler

Request dispatch is formalized as a parallel‑machine deadline‑scheduling problem. Janus proposes the LST‑IMH algorithm, which extends the Moore–Hodgson single‑machine approach to multiple instances, prioritizing requests that can meet their deadlines. LST‑IMH guarantees a 2‑approximation for maximizing on‑time completions with time complexity O(m n log n).

Engine Layer – xTensor

xTensor exposes HBM as a unified virtual address space and organizes physical pages into a single page pool. Model weights grow from low addresses, while KV cache and activation values grow from high addresses, enabling bidirectional allocation, dynamic sharing, and low fragmentation. The global address region also serves as the receive buffer for D2D Fork, avoiding host‑to‑device transfers.

Three‑State Model Lifecycle & D2D Fork

Models transition among Active (weights and runtime ready), Warm (runtime state retained, some devices released), and Cold (all device resources released). Recovery can use host‑side H2D Wakeup or intra‑interconnect D2D Fork, which copies weights directly between active replicas, reducing activation latency from seconds to sub‑second for large models.

Experimental Results

Janus is deployed on a production cluster of 768 accelerators serving 62 applications (model sizes 0.6 B–671 B) with ~45.6 M daily requests. In an open‑loop replay of 603 k requests, Janus achieves an SLO fulfillment rate of 0.97–1.0, compared with a baseline of 0.80–0.92, while using 13 440 device‑hours daily—a 27 % reduction versus a static ServerlessLLM baseline.

Against Prism, BlitzScale, and ServerlessLLM, Janus sustains 1.3–1.5× higher request rates at a 90 % SLO target and meets latency targets that are five times stricter. Ablation of the dual‑pool design shows overall SLO 97.8 % (vs. 59.0 % for Elastic‑only and 37.1 % for Steady‑only) in a 120‑second mixed‑load experiment.

D2D Fork accelerates activation of a 671 B DeepSeek‑V3 model to 0.74 s versus 2.71 s for H2D Wakeup (≈3.7× faster) and maintains ~98 % SLO during bursts. xTensor enables three‑fold model density (one device vs. three for static partitioning) and keeps fragmentation below 0.3 % under sustained allocation pressure.

Conclusion

Janus addresses bursty traffic, power‑law hotness, and heterogeneous resource needs through dual‑timescale scheduling, coordinated Service‑Engine design, and runtime mechanisms (xTensor, D2D Fork, three‑state lifecycle). It reduces device usage by 27 % while keeping SLO rates at 0.97–1.0, demonstrating that production‑grade multi‑LLM serving requires both global scheduling coordination and low‑level resource management.