RSI-Nav: Self-Improving Robot Navigation with Unlabeled Trajectories, +17.7% Success

Problem: Scarcity of Labeled Instruction-Trajectory Pairs

Vision-Language Navigation (VLN) requires paired natural-language instructions and navigation trajectories. While trajectories can be collected at scale in simulation, writing reliable instructions for each trajectory is labor-intensive. Existing methods use separate models for instruction generation (VA2L) and navigation (VL2A), and often employ a static data flywheel: a fixed generator labels a large offline dataset once, then a navigation model is trained on the synthetic data. This creates a policy gap — as the navigation policy improves, the generated instructions remain tied to the old generator, limiting further gains.

RSI-Nav: Unified Model and Recursive Self-Improvement

Researchers from Fudan University propose RSI-Nav (Data-Centric Recursive Self-Improvement for Navigation) , built on Qwen3.5-9B . A single model handles three tasks sharing all parameters:

VL2A (Vision-Language-to-Action) : given an instruction and egocentric RGB history, predict actions (forward, left, right, stop).

VA2L (Vision-Action-to-Language) : given a trajectory's visual observations and action sequence, generate a natural-language instruction.

Spatial Understanding : learn object locations, relative directions, and spatial relations.

The key insight is that learning to describe a path forces the model to discriminate landmarks and turning points, which in turn provides richer supervision for following a path .

Policy-Synchronized Language-Action Dual Bootstrapping

RSI-Nav replaces the static flywheel with a tight Generate → Execute → Filter → Update loop:

Generate : the current model reads a batch of unlabeled visual-action trajectories and writes navigation instructions.

Execute : the same model starts from the original trajectory's start point and follows the newly generated instruction in the simulator.

Filter : keep only samples where the model reaches the target vicinity and the executed path is sufficiently efficient (success + path-efficiency criteria).

Update : immediately fine-tune the model on the filtered pairs (using the original reference trajectory as action supervision, not the executed path) plus spatial-understanding data, then proceed to the next batch.

Batches contain 16,384 trajectories ; the model is updated after each batch. This policy-synchronized approach (not strictly on-policy, because source trajectories remain from an offline pool) ensures generated instructions and verification reflect the model's current capability. The whole process uses supervised fine-tuning (SFT) without reinforcement learning.

After self-improvement, RSI-Nav yields ~2.5 million navigation samples and ~1.1 million instruction-generation samples .

Benchmark Results on VLN-CE (Val-Unseen)

Evaluated on continuous VLN benchmarks R2R-CE and RxR-CE using only egocentric RGB (no depth, odometry, or external waypoint predictor):

R2R-CE success rate : 68.6% (Cold Start 52.9% → +15.7 pp )

RxR-CE success rate : 72.4% (Cold Start 54.7% → +17.7 pp )

SPL and nDTW also improve, indicating better path efficiency and instruction adherence.

Ablation: Where Gains Come From

Navigation data only: 44.7% success.

+ VA2L (trajectory-to-instruction) supervision: 48.8%.

+ Spatial understanding data: 52.9%.

This confirms that reverse language-action supervision and spatial reasoning both contribute.

Instruction Quality Measured by Downstream Navigation

Beyond text similarity, the authors train identical downstream navigators on instructions from different generators. Using RSI-Nav instructions yields 41.3% success / 35.1% SPL , versus 36.8% / 30.4% for SRDF-generated instructions — showing synthetic instructions are valuable because they provide effective navigation supervision , not just fluent text.

Update Frequency Ablation

Moderate batch-wise updates (16,384 trajectories per update) outperform both very frequent updates and a single full-dataset offline update.

Real-World Deployment on DEEP Robotics M20

The model is deployed on a DEEP Robotics M20 quadruped with inference on a remote NVIDIA RTX 4090 . The robot receives natural-language commands and egocentric RGB, outputting continuous navigation actions. Demonstrations include indoor obstacle avoidance, narrow corridors, multi-step instructions, open-vocabulary landmark recognition, cross-room long-horizon routes, and outdoor navigation. Example: "from the plant, go through the door, find the quadruped robot, enter the bright room, stop at the washing machine." The paper notes these are qualitative real-robot demonstrations , not large-scale success-rate statistics; further testing in dynamic, open environments is needed.

Conclusion

RSI-Nav shows that a single vision-language model can recursively improve its own navigation and instruction-generation capabilities by tightly coupling data generation, execution verification, and model updates. The approach moves beyond simply scaling synthetic data volume to synchronizing data quality with the evolving policy, and it transfers to a physical quadruped without additional fine-tuning.