PhyAI: The First Unified Edge‑Cloud Inference Runtime for Physical AI

01 Deployment Scenarios and Problem Definition

Physical AI inference includes four typical scenarios: benchmark evaluation (accuracy focus), cloud reinforcement‑learning rollout (high‑throughput, multi‑GPU utilization), edge deployment (latency‑critical), and factory MaaS where a shared GPU server serves multiple robots (network, queuing, and batching affect latency). Batch size, model precision, and hardware differ, but image preprocessing, inference logic, caching, and action output must be reusable and consistent.

Existing pipelines maintain separate codebases for each scenario, causing high migration cost, duplicated engineering effort, and inconsistent inference efficiency.

02 Unified Inference Runtime Design

PhyAI provides a single runtime that can execute the same model path on a single GPU, edge devices, or multi‑GPU cloud servers. It consists of two components:

Model Runner : stores visual‑language conditions, action expert or video‑action generation, solver, state reuse, and action conversion.

Scheduler : selects data‑parallel (DP), tensor‑parallel (TP), CFG, and device groups.

The Runner manages KV cache, buffers, CUDA Graph, and request state. Layers choose fused or distributed operators based on tensor shape, dtype, and hardware. This eliminates the need for four separate code paths.

03 Inference Bottlenecks and Control‑Time Roofline

Component‑wise measurements show model‑side bottlenecks. For model PI0.5 at batch=1, the action expert accounts for 8.8% of FLOPs but 57.2% of latency; at batch=32 its share drops to 13.5% while throughput reaches ≈100 samples/s. For Cosmos3, increasing batch from 1 to 16 improves throughput by only 14.3%, indicating a compute‑bound regime.

Real‑Time Chunking, network delay, and image preprocessing/post‑processing prevent the model‑side Roofline from directly predicting robot control rates. The authors introduce the Control‑Time Roofline to quantify whether control latency is dominated by inference or physical‑environment factors. Tests on an AGX Orin show inference as the primary bottleneck, whereas on an RTX Pro 6000 the environment dominates.

The analysis concludes that on compute‑rich devices further latency reduction yields diminishing returns for control frequency; algorithm, hardware, and infrastructure must be co‑designed, and saved time should support larger models, slower devices, or higher concurrency.

04 Benchmark and RL Rollout

In 11 paired single‑request experiments (same model, same device), PhyAI outperforms the official implementation by 1.40×–4.65×. For MiniCPM‑Robot on an H100 GPU, latency drops from 105.38 ms to 22.64 ms. For Cosmos3‑Nano‑Policy‑DROID on 8 × H20 GPUs (CFG=2, TP=4), latency falls from 2.46 s to 1.18 s.

In a cloud RL rollout (RLinf) using PI0.5 GRPO on 4 × A800 GPUs with 32 environments, inference consumes 955.8 s (15.9% of total RL time). PhyAI’s 2.55× inference speedup reduces inferred time to 863.2 s, cutting overall RL inference by 9.7% while keeping other components unchanged.

05 Conclusion

PhyAI unifies benchmark, cloud RL rollout, edge deployment, and factory MaaS inference within a single runtime, removing duplicated model adapters, operator optimizations, and multi‑GPU support across four separate paths. The Control‑Time Roofline clarifies that robot performance is jointly constrained by inference speed and environmental factors, and that accelerating inference alone does not linearly increase control frequency.

Paper: https://arxiv.org/abs/2608.03682

Code: https://github.com/mingti-org/phyai