AgentVLN: VLM-as-Brain Architecture for Agentic Robot Navigation at ECCV 2026

Overview

AgentVLN is a vision-language navigation (VLN) system accepted at ECCV 2026 that adopts a "VLM-as-Brain" paradigm. Instead of relying on a monolithic end-to-end model, a vision-language model (VLM) handles high-level cognitive tasks — understanding instructions, analyzing the environment, and deciding which skills to invoke — while a modular skill library executes low-level functions such as SLAM, perception, obstacle avoidance, and motion control. The pipeline follows: VLM Agent → Skill Calling → SLAM/Perception → Robot Navigation.

Core Idea: VLM as Brain with Modular Skills

The navigation process is modeled as a partially observable semi-Markov decision process. Skills are divided into perception-layer skills (mapping, localization, traversable area detection, candidate waypoint generation) and planning-layer skills (path selection, exploration). This design mirrors human navigation: perceive environment → reason about route → act → adjust based on feedback. The same VLM "brain" can be paired with different low-level skill modules to adapt to various robot platforms without retraining the VLM.

See environment → Think route → Act → Adjust based on feedback.

A plugin-style skill library enables adaptation to new environments and tasks by extending or swapping skills, without retraining the large VLM.

Cross-Space Representation Mapping

VLMs excel at 2D image semantics, but robot navigation operates in 3D. AgentVLN bridges this gap with a Cross-Space Representation Mapping mechanism:

The robot invokes perception skills to build a 3D map, extract topology, compute traversable regions, and generate candidate 3D navigation waypoints.

Using the camera model, these 3D waypoints are projected into the current 2D camera image, producing pixel-aligned visual prompts. The VLM then selects the waypoint that best matches the language instruction, turning an open-ended 3D coordinate regression problem into a choice-based selection in visual space.

"Select the waypoint in the image that best matches the language instruction."

This leverages the VLM's strong visual-semantic understanding while avoiding direct complex 3D geometric reasoning. The selected 2D waypoint is then mapped back to 3D for navigation control.

Context-Driven Self-Correction

Real-world deployments face occlusions, lighting changes, and sensor blind spots. AgentVLN incorporates a self-correction mechanism: when the current path becomes invisible or the environment mismatches the instruction, the agent actively executes exploratory actions — turning, reorienting, searching for new paths — to prevent error accumulation during long-horizon navigation.

Query-Driven Perceptual Chain-of-Thought (QD-PCoT)

To address depth ambiguity in 2D vision, AgentVLN introduces QD-PCoT. When the model cannot confidently locate a target (e.g., distance to a chair), it actively poses a question ("How far is the chair ahead?") and calls the appropriate perception skill (depth estimation) to acquire the missing spatial information. The process follows: Active Question → Information Retrieval → Belief Update → Decision Making. Experiments show QD-PCoT improves navigation performance without adding parameters, mitigating depth ambiguity in 2D visual inputs.

Active questioning → Get information → Update cognition → Decide action.

Edge Deployment with a 3B Model

AgentVLN uses Qwen2.5-VL-3B as its backbone, avoiding heavy 3D vision modules. The lightweight design enables real-time inference on NVIDIA Jetson edge devices. The system has been deployed on quadruped robot dogs and humanoid platforms, where onboard cameras feed perception skills for mapping and path planning.

Results and Resources

Paper: AgentVLN: Towards Agentic Vision-and-Language Navigation (ECCV 2026)

Code: https://github.com/Allenxinn/AgentVLN

Project page: https://allenxinn.github.io/AgentVLN/

Dataset: https://huggingface.co/datasets/allenxinn/AgentVLN-Instruct

AgentVLN achieves leading performance on R2R-CE and RxR-CE benchmarks while maintaining a compact 3B parameter size, demonstrating that a VLM-driven modular architecture can bring agentic navigation to real robots at the edge.