Jetson-PI Enables Real‑Time VLA on Robots, Boosting Jetson Orin Control Frequency 8.66×

Problem

Deploying multi‑billion‑parameter Vision‑Language‑Action (VLA) models on mobile robots faces two critical obstacles: (1) high power consumption of external GPUs limits battery life and operational range, and (2) inference latency on edge devices such as NVIDIA Jetson Orin leads to low control frequencies (≈0.7 Hz) and perceptual‑action misalignment.

Solution Overview (Jetson‑PI)

Jetson‑PI addresses the problem from three complementary angles: asynchronous inference with future correction, confidence‑based scheduling, and a lightweight edge inference engine built on llama.cpp.

1. Foresight‑Aligned Asynchronous Correction

The system introduces a future‑correction module that receives (i) the current visual‑language model (VLM) representation and (ii) the already‑issued action sequence. It predicts the VLM representation at the future time when the pending actions finish, allowing the action expert to generate actions aligned with that future state. The module compresses only the last VLM layer, contains ≈40 M parameters (≈1 % of the full VLA model), and is trained with randomly sampled future time spans so it can adapt to varying hardware delays.

2. Confidence‑Based Scheduling

The future‑correction module also outputs a confidence score. When the score exceeds a preset threshold, the system skips a fresh VLM observation and directly invokes the action expert with the corrected representation, reducing the VLM call frequency. If confidence falls below the threshold—typically during grasping, contact, or placement phases—the VLM is re‑executed to refresh the environment representation, KV cache, and subsequent predictions.

3. Jetson‑PI‑Edge Engine Optimizations

Implemented on llama.cpp, the engine applies three low‑level optimizations tailored to VLA workloads:

CUDA‑graph reuse : Fixed input dimensions (camera count, image resolution, action dimensionality) allow the first inference to construct a CUDA graph that is reused for all subsequent control steps, eliminating per‑step graph construction.

Resident intermediate caches : Visual embeddings and the language model KV cache are allocated in persistent GPU buffers, avoiding costly CPU‑GPU transfers between modules.

Flow‑matching expansion : Multi‑step denoising in the action expert is unrolled into a single CUDA graph, removing repeated kernel‑launch overhead.

These optimizations do not alter the model computation and can be combined with quantization or pruning.

Performance Results

On an NVIDIA Jetson Orin, the π0.5 model achieves a single‑inference latency of 412.9 ms (down from 1.4 s), raising the control frequency to 6.06 Hz** (reaction time 165.1 ms)**—an 8.66× speedup over native PyTorch. The action expert’s latency drops from 536.8 ms to 123.1 ms after applying the three engine optimizations.

Benchmarking on the LIBERO suite (Spatial, Object, Goal, LIBERO‑10) shows:

Average success‑rate improvement of 14.8 percentage points over VLASH and 3.9 points over RTC across all asynchronous delay settings.

At a large delay (Δ = 9), Jetson‑PI’s average success rate exceeds VLASH by 45.6 points and RTC by 7.0 points, demonstrating robustness to latency.

Real‑world experiments on the PrimeBot X2‑W robot performing a 15 Hz clothing‑folding task exhibit continuous, smooth motions without the pauses observed in synchronous or naïve asynchronous baselines. The robot uses one head camera and two wrist cameras (224 × 224 resolution) and maintains high success rates through the confidence‑driven scheduling mechanism.

Resources

Paper: Jetson‑PI: Towards Onboard Real‑Time Robot Control via Foresight‑Aligned Asynchronous Inference (arXiv:2607.12659)

Code repositories: https://github.com/PKU-SEC-Lab/Jetson-PI (asynchronous inference implementation) https://github.com/PKU-SEC-Lab/Jetson-PI-Edge (edge inference engine)