Skip to content
NEO-VLA: Native VLA Model Generates Robot Actions Directly from Pixels

NEO-VLA: Native VLA Model Generates Robot Actions Directly from Pixels

Researchers from NTU, SenseTime, and Ropedia propose NEO-VLA, a native vision-language-action model that eliminates separate vision encoders by feeding images through lightweight convolutions into a multimodal backbone, allowing the action expert to directly access multi-layer visual representations for more precise robot manipulation, validated across simulations and real-world tasks.

Machine Heart
Machine Heart
Machine Heart

Introduction

Precise robot manipulation such as placing a tennis ball on a can requires fine-grained visual cues — object edges, local geometry, depth, and subtle gripper-object state changes. Most existing vision-language-action (VLA) models first pass images through a pre-trained vision encoder (CLIP, SigLIP, DINOv2) and then feed the encoded features to a language backbone and action head. However, as visual information is progressively abstracted across encoder layers, manipulation-relevant details may become harder to recover.

NEO-VLA Architecture

NEO-VLA introduces a native vision-language backbone (NEO-2B, 40 layers, ~2B parameters) with no independent vision Transformer, autoencoder, or discrete visual codebook. Images are processed by a lightweight convolutional stem and then jointly modeled with language instructions from the first backbone layer. A separately initialized action expert reads cached key-value representations from specified backbone layers via a configurable layer-mapping scheme.

Figure 1: Visual interfaces and action output methods of three VLA designs
Figure 1: Visual interfaces and action output methods of three VLA designs
Figure 2: NEO-VLA multimodal input, backbone, and action expert
Figure 2: NEO-VLA multimodal input, backbone, and action expert

Layer Mapping Schemes

The action expert's layers are mapped to backbone layers. Four schemes are compared:

first-28 : reads layers 0–27, backbone truncated to 28 layers.

skip-28 : keeps 40-layer backbone, samples 28 layers approximately uniformly.

end-28 : keeps 40-layer backbone, reads layers 12–39.

full-40 : reads all 40 layers with a 40-layer action expert.

The first three use a 28-layer action expert; skip-28 and end-28 share backbone depth but differ in readout depth, isolating the effect of shallow vs. deep representations.

Action Generation via Flow Matching

The action expert uses flow matching to predict a 30-step action chunk covering both arms and body in one forward pass. At inference, 10-step Euler integration is used; the backbone runs once and its KV cache is reused across sampling steps.

Three-Stage Training

Stage 1 (freeze backbone) : train action expert and condition encoder for 113,400 steps, global batch size 8,192.

Stage 2 (unfreeze all) : continue on same robot data for 75,000 steps, interleaving one multimodal understanding batch every other optimization step to preserve vision-language capabilities.

Stage 3 (internal data) : further 75,000 steps of full-parameter training on proprietary data, same configuration as Stage 2.

Unified Dataset and Action Format

Pre-training corpus aggregates 2,231,583 trajectories (~928M records, ~10k hours) from nine public datasets (AgiBot-World, Bimanual YAM, RoboCOIN, Open X-Embodiment, Galaxea Open-World, RoboMIND, RH20T, DROID, InternData-A1) covering 2,704 scenes and 7,444 camera views. Actions are normalized per robot embodiment into a unified 60-dim vector (26 per arm: 7 joint targets, 3 translation deltas, 3 rotation deltas, 1 gripper, 12 hand joints; 8 for body: 3 base, 3 torso, 2 head) with validity masks for missing channels.

Five-tier quality checks (reading, channel, action target, trajectory, calibration) handle anomalies at the appropriate granularity without discarding entire trajectories.

Figure 3: Five data quality checks and processing scopes
Figure 3: Five data quality checks and processing scopes

Key Experimental Findings

Native Backbone Effectiveness

Controlled comparison on Open X-Embodiment pre-training (75k steps) and LIBERO evaluation with frozen backbones: NEO-VLA achieves 90.9% average success, outperforming Qwen3-VL-2B (79.5%) and Gemma-3-4B (84.5%). NEO-VLA frozen even matches or exceeds the fully fine-tuned baselines (90.8% and 91.0%). On LIBERO-Long, frozen NEO-VLA reaches 85.4%, surpassing both fine-tuned baselines. This indicates NEO's pre-trained representations transfer well to action tasks.

An ablation on Qwen3-VL connects its action expert directly to the vision encoder's intermediate features (no extra params). Frozen backbone performance drops from 79.5% to 64.9%, while fine-tuned performance converges (~90.9% vs 90.8%). Thus, merely accessing intermediate features is insufficient; the way representations are learned (native joint pre-training) matters.

Table 2: Controlled backbone comparison on LIBERO
Table 2: Controlled backbone comparison on LIBERO
Table 3: Qwen3-VL with direct vision intermediate feature access
Table 3: Qwen3-VL with direct vision intermediate feature access

Shallow Layer Sensitivity

Layer-wise intervention: replacing KV at layers 0–10 with another observation's KV causes the largest translation prediction error increase; matched-magnitude noise adds <0.004 cm error, confirming sensitivity is content-driven. Freezing the backbone makes shallow representations especially critical; full fine-tuning benefits from multi-depth fusion. Language, appearance, and geometry information exhibit distinct depth distributions.

Figure 4: Representation replacement and input masking responses across layers
Figure 4: Representation replacement and input masking responses across layers
Figure 4 continued
Figure 4 continued

Joint Understanding-Action Training

Comparing post-training with only action data vs. joint action+understanding data (interleaved). Joint training maintains understanding benchmarks (MMBench-EN, RealWorldQA, AI2D, TextVQA, ChartQA) within 1.5% drop (RealWorldQA +1.6%) and achieves 69.4% on RoboTwin 2.0 Clean2Rand. Action-only training destroys text generation (benchmarks marked "—") and drops Clean2Rand to 10.7%. Multimodal understanding is tightly coupled with robust action adaptation.

Table 4: Joint vs. action-only post-training comparison
Table 4: Joint vs. action-only post-training comparison

Benchmark Evaluations

LIBERO

Full NEO-VLA model scores 97.4% average success on LIBERO, indicating strong in-distribution manipulation capability.

Table 5: Full model LIBERO results
Table 5: Full model LIBERO results

LIBERO-Plus (Robustness)

Seven perturbation categories: NEO-VLA adapts well to observation and image condition changes (consistent with shallow-layer analysis), but embodiment and spatial configuration changes remain challenging.

Table 6: LIBERO-Plus perturbation results
Table 6: LIBERO-Plus perturbation results

LIBERO-Pro (Generalization)

Instruction semantics change: 98.0%; object appearance change: 84.3%; object position and task goal changes show poor performance across methods, revealing a gap between perception robustness and flexible spatial/task adaptation.

Table 7: LIBERO-Pro generalization results
Table 7: LIBERO-Pro generalization results

RoboTwin 2.0

On 2,500 demonstration trajectories: Clean 75.78%, Clean2Rand 69.40% (6.38% drop), demonstrating reasonable sim-to-real transfer under randomization.

Table 8: RoboTwin 2.0 Clean vs. Clean2Rand
Table 8: RoboTwin 2.0 Clean vs. Clean2Rand

Real-World Experiments

Three tasks (precise placement, multi-step operation, object pose adjustment) each fine-tuned with 50 real demonstrations, tested 20 times (10 in-distribution, 10 out-of-distribution). NEO-VLA succeeds 12, 17, 13 times respectively; StarVLA: 1, 2, 0; π0.5: 10, 14, 13. NEO-VLA matches or exceeds π0.5 on all tasks. Additional demos include pouring and picking sushi pieces.

Figure 5: Three real-world quantitative evaluation tasks
Figure 5: Three real-world quantitative evaluation tasks
Table 9: Real-world task success counts (20 trials each)
Table 9: Real-world task success counts (20 trials each)
Figure 6: Real-world operation demos and key motion sequences
Figure 6: Real-world operation demos and key motion sequences

Inference Latency

On a single RTX 3090 (24GB, BF16), the model returns a 30-step action chunk in ~0.26 seconds (measured on Pika real-robot system, no RTC).

Table 10: Model forward and real-robot request latency in milliseconds
Table 10: Model forward and real-robot request latency in milliseconds

Conclusion

NEO-VLA demonstrates that action learning benefits from both how visual representations are learned (native joint vision-language pre-training) and how the action module accesses them (multi-layer readout). Frozen native backbone already supports strong action transfer; shallow-layer access is critical when the backbone is frozen, while full fine-tuning leverages deeper layers. Simply exposing intermediate features of a traditional vision encoder does not replicate the gain. Joint understanding-action post-training preserves semantic capabilities and improves randomization robustness. Remaining challenges include spatial/task generalization under novel configurations — future work must translate perception and understanding into more flexible action adaptation.

Original Source

Signed-in readers can open the original source through BestHub's protected redirect.

Sign in to view source
Republication Notice

This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactadmin@besthub.devand we will review it promptly.

Embodied AIFlow MatchingRobot ManipulationVision-Language-Action ModelMulti-layer Visual RepresentationsNative Multimodal BackboneNEO-VLARobot Action Learning
Reader feedback

How this landed with the community

Sign in to like

Rate this article

Was this worth your time?

Sign in to rate
Discussion

0 Comments

Thoughtful readers leave field notes, pushback, and hard-won operational detail here.