LCVN: Predicting Full Navigation Paths from One Frame and Language Instruction

Task Definition

LCVN (Language-Conditioned Visual Navigation) requires predicting a full navigation trajectory — continuous 2D displacements and heading changes — from only a single first-person starting image and a natural-language instruction. Unlike goal-conditioned visual navigation that provides a target image, LCVN uses language to describe the goal and intermediate landmarks. A critical constraint: during prediction the model receives no real-time environment feedback for correction; it must rely solely on its own predicted states, making offline trajectory prediction the core challenge.

Dataset

The LCVN dataset comprises 39,016 trajectories and 117,048 human-verified instructions sourced from five existing datasets: Go Stanford, ReCon, SCAND, HuRoN (used for training and in-domain evaluation), and TartanDrive (off-road scenes for out-of-domain generalization). Each trajectory is paired with three English instruction styles: concise (main directional cues), detailed (rich scene descriptions), and landmark (route organized around reference objects). Instructions were generated by a model conditioned on first-person video and top-down trajectory maps, then manually verified. A human evaluation of 640 random trajectories gave an overall quality score of 5.2/6.0.

Two Approaches: Combining Imagination and Decision-Making

1. LCVN-WM + LCVN-AC: Policy Learning in an Imagined World

This pipeline separates a world model (LCVN-WM) and a policy model (LCVN-AC).

LCVN-WM extracts image features and, conditioned on history states, actions, and the language instruction (via cross-attention into a diffusion model), predicts future scene changes. Training uses Diffusion Forcing , adding varying noise levels to different context states to strengthen continuous prediction.

LCVN-AC learns a policy entirely in the world model's latent space: it selects actions, the world model predicts resulting state changes, and rewards are derived from proximity to expert trajectories. The model also learns to infer latent plans from the current state and instruction, aligning them with plans extracted from full expert trajectories.

The world model directly participates in policy optimization; its predicted future states influence how actions are refined.

2. LCVN-Uni: Joint Action and Frame Prediction in a Unified Model

LCVN-Uni builds on the Anole-7B multimodal autoregressive model. It tokenizes images, language, and discretized actions into a single sequence. The model jointly predicts the next action and the next observation frame in one forward pass, optimizing both prediction targets simultaneously. The generated frame becomes the input for the next step, rolling out the entire trajectory.

The key distinction: the first approach trains a separate policy inside a world model; the second lets a single model learn action and visual prediction together.

Experimental Results

Evaluation metrics: navigation success rate (predicted endpoint within one average step length of true goal), trajectory error, and predicted frame quality (SSIM, DreamSim).

In-Domain Validation

LCVN-WM + LCVN-AC: 43% success rate

LCVN-Uni: 42%

NWM + LCVN-AC (world model without language): 29%

Out-of-Domain Generalization (TartanDrive)

LCVN-Uni: 27% success rate

LCVN-WM + LCVN-AC: 19%

Mixed Test Set (RoboWorld Challenge WorldNav Track)

LCVN-WM + LCVN-AC: 36%

LCVN-Uni: 34%

Frame Prediction Quality

LCVN-WM maintains better structural and temporal coherence: SSIM@8 = 0.293 vs. LCVN-Uni's 0.218 . However, LCVN-Uni scores higher on DreamSim, indicating better perceptual similarity. This shows visual prediction capability and navigation capability must be assessed separately; no single metric captures overall performance.

Language Ablation

Removing language conditioning degrades both navigation and frame prediction. For LCVN-WM + LCVN-AC, in-domain success drops from 43% to 35% . Qualitative examples (e.g., instruction "proceed along corridor until you see orange wall on right") show that language helps preserve goal-relevant scene cues during long-horizon prediction, stabilizing subsequent action decisions.

Limitations and Future Work

A failure case: LCVN-Uni correctly predicts turn direction but fails to generate the post-turn corridor and target white door, leaving subsequent actions without reliable visual grounding. Remaining challenges: real-robot closed-loop validation (currently only offline prediction), instruction diversity beyond three preset English styles, natural spoken language, multilingual instructions, and recovery from execution errors.

Resources

Paper: https://arxiv.org/abs/2603.26741

Code: https://github.com/UWMILab/LCVN

Project page: https://uwmilab.github.io/LCVN-webpage/

Dataset: https://huggingface.co/datasets/fly1113/LCVN

Competition: https://roboworld2026.github.io/track1

Leaderboard: https://www.codabench.org/competitions/18185/