NEO-VLA: Native VLA Model Generates Robot Actions Directly from Pixels
Researchers from NTU, SenseTime, and Ropedia propose NEO-VLA, a native vision-language-action model that eliminates separate vision encoders by feeding images through lightweight convolutions into a multimodal backbone, allowing the action expert to directly access multi-layer visual representations for more precise robot manipulation, validated across simulations and real-world tasks.
Introduction
Precise robot manipulation such as placing a tennis ball on a can requires fine-grained visual cues — object edges, local geometry, depth, and subtle gripper-object state changes. Most existing vision-language-action (VLA) models first pass images through a pre-trained vision encoder (CLIP, SigLIP, DINOv2) and then feed the encoded features to a language backbone and action head. However, as visual information is progressively abstracted across encoder layers, manipulation-relevant details may become harder to recover.
NEO-VLA Architecture
NEO-VLA introduces a native vision-language backbone (NEO-2B, 40 layers, ~2B parameters) with no independent vision Transformer, autoencoder, or discrete visual codebook. Images are processed by a lightweight convolutional stem and then jointly modeled with language instructions from the first backbone layer. A separately initialized action expert reads cached key-value representations from specified backbone layers via a configurable layer-mapping scheme.
Layer Mapping Schemes
The action expert's layers are mapped to backbone layers. Four schemes are compared:
first-28 : reads layers 0–27, backbone truncated to 28 layers.
skip-28 : keeps 40-layer backbone, samples 28 layers approximately uniformly.
end-28 : keeps 40-layer backbone, reads layers 12–39.
full-40 : reads all 40 layers with a 40-layer action expert.
The first three use a 28-layer action expert; skip-28 and end-28 share backbone depth but differ in readout depth, isolating the effect of shallow vs. deep representations.
Action Generation via Flow Matching
The action expert uses flow matching to predict a 30-step action chunk covering both arms and body in one forward pass. At inference, 10-step Euler integration is used; the backbone runs once and its KV cache is reused across sampling steps.
Three-Stage Training
Stage 1 (freeze backbone) : train action expert and condition encoder for 113,400 steps, global batch size 8,192.
Stage 2 (unfreeze all) : continue on same robot data for 75,000 steps, interleaving one multimodal understanding batch every other optimization step to preserve vision-language capabilities.
Stage 3 (internal data) : further 75,000 steps of full-parameter training on proprietary data, same configuration as Stage 2.
Unified Dataset and Action Format
Pre-training corpus aggregates 2,231,583 trajectories (~928M records, ~10k hours) from nine public datasets (AgiBot-World, Bimanual YAM, RoboCOIN, Open X-Embodiment, Galaxea Open-World, RoboMIND, RH20T, DROID, InternData-A1) covering 2,704 scenes and 7,444 camera views. Actions are normalized per robot embodiment into a unified 60-dim vector (26 per arm: 7 joint targets, 3 translation deltas, 3 rotation deltas, 1 gripper, 12 hand joints; 8 for body: 3 base, 3 torso, 2 head) with validity masks for missing channels.
Five-tier quality checks (reading, channel, action target, trajectory, calibration) handle anomalies at the appropriate granularity without discarding entire trajectories.
Key Experimental Findings
Native Backbone Effectiveness
Controlled comparison on Open X-Embodiment pre-training (75k steps) and LIBERO evaluation with frozen backbones: NEO-VLA achieves 90.9% average success, outperforming Qwen3-VL-2B (79.5%) and Gemma-3-4B (84.5%). NEO-VLA frozen even matches or exceeds the fully fine-tuned baselines (90.8% and 91.0%). On LIBERO-Long, frozen NEO-VLA reaches 85.4%, surpassing both fine-tuned baselines. This indicates NEO's pre-trained representations transfer well to action tasks.
An ablation on Qwen3-VL connects its action expert directly to the vision encoder's intermediate features (no extra params). Frozen backbone performance drops from 79.5% to 64.9%, while fine-tuned performance converges (~90.9% vs 90.8%). Thus, merely accessing intermediate features is insufficient; the way representations are learned (native joint pre-training) matters.
Shallow Layer Sensitivity
Layer-wise intervention: replacing KV at layers 0–10 with another observation's KV causes the largest translation prediction error increase; matched-magnitude noise adds <0.004 cm error, confirming sensitivity is content-driven. Freezing the backbone makes shallow representations especially critical; full fine-tuning benefits from multi-depth fusion. Language, appearance, and geometry information exhibit distinct depth distributions.
Joint Understanding-Action Training
Comparing post-training with only action data vs. joint action+understanding data (interleaved). Joint training maintains understanding benchmarks (MMBench-EN, RealWorldQA, AI2D, TextVQA, ChartQA) within 1.5% drop (RealWorldQA +1.6%) and achieves 69.4% on RoboTwin 2.0 Clean2Rand. Action-only training destroys text generation (benchmarks marked "—") and drops Clean2Rand to 10.7%. Multimodal understanding is tightly coupled with robust action adaptation.
Benchmark Evaluations
LIBERO
Full NEO-VLA model scores 97.4% average success on LIBERO, indicating strong in-distribution manipulation capability.
LIBERO-Plus (Robustness)
Seven perturbation categories: NEO-VLA adapts well to observation and image condition changes (consistent with shallow-layer analysis), but embodiment and spatial configuration changes remain challenging.
LIBERO-Pro (Generalization)
Instruction semantics change: 98.0%; object appearance change: 84.3%; object position and task goal changes show poor performance across methods, revealing a gap between perception robustness and flexible spatial/task adaptation.
RoboTwin 2.0
On 2,500 demonstration trajectories: Clean 75.78%, Clean2Rand 69.40% (6.38% drop), demonstrating reasonable sim-to-real transfer under randomization.
Real-World Experiments
Three tasks (precise placement, multi-step operation, object pose adjustment) each fine-tuned with 50 real demonstrations, tested 20 times (10 in-distribution, 10 out-of-distribution). NEO-VLA succeeds 12, 17, 13 times respectively; StarVLA: 1, 2, 0; π0.5: 10, 14, 13. NEO-VLA matches or exceeds π0.5 on all tasks. Additional demos include pouring and picking sushi pieces.
Inference Latency
On a single RTX 3090 (24GB, BF16), the model returns a 30-step action chunk in ~0.26 seconds (measured on Pika real-robot system, no RTC).
Conclusion
NEO-VLA demonstrates that action learning benefits from both how visual representations are learned (native joint vision-language pre-training) and how the action module accesses them (multi-layer readout). Frozen native backbone already supports strong action transfer; shallow-layer access is critical when the backbone is frozen, while full fine-tuning leverages deeper layers. Simply exposing intermediate features of a traditional vision encoder does not replicate the gain. Joint understanding-action post-training preserves semantic capabilities and improves randomization robustness. Remaining challenges include spatial/task generalization under novel configurations — future work must translate perception and understanding into more flexible action adaptation.
Signed-in readers can open the original source through BestHub's protected redirect.
This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactand we will review it promptly.
How this landed with the community
Was this worth your time?
0 Comments
Thoughtful readers leave field notes, pushback, and hard-won operational detail here.
