HA-VLN 2.0: Benchmarking Socially-Aware Navigation in Dynamic Human Environments

Introduction

Navigation instructions such as "go up the stairs, pass the person on the phone, then cross the bedroom" become challenging in populated environments: people pace while talking, chairs require clearance, and doorways may be blocked by conversing groups. Robots must understand the route and continuously adapt to observed changes.

Researchers from the University of Washington, Carnegie Mellon University, National University of Singapore, and Microsoft Research propose HA-VLN 2.0 (Human-Aware Vision-and-Language Navigation), accepted at IROS 2026. This work extends human-aware VLN into an open benchmark covering both discrete and continuous environments, integrating dynamic humans, multi-person interactions, and social constraints into a unified research framework.

HA-VLN 2.0 Benchmark: Discrete and Continuous Environments

HA-VLN 2.0 supports two modes:

Discrete Environment (DE) : agent selects next panoramic viewpoint; suitable for large-scale testing.

Continuous Environment (CE) : agent executes fine-grained forward/turn actions; directly exposes obstacle avoidance, turning, and human-approach behaviors.

Both modes share human-aware tasks, data resources, and interfaces, but action execution and collision checking retain separate settings. The paper analyzes step length and collision-check frequency effects to avoid mistaking coarser viewpoint movement for better avoidance capability.

HAPS 2.0: Human Activity Dataset for Realistic Scene Population

The benchmark builds on the upgraded HAPS 2.0 human activity resource. Core statistics:

172 activity categories

486 motion sequences

58,320 motion frames

26 region types

After placement and interaction augmentation, scenes contain 910 annotated human instances distributed across 90 building scans and 428 regions . These numbers describe action categories, motion sequences, and scene instances respectively and are not interchangeable. Placement uses Particle Swarm Optimization (PSO) for coarse positioning, followed by nine-view checks for penetration, posture, and spatial relations, plus manual review for multi-person interactions.

The dataset captures diverse indoor/outdoor activities: walking, sudden stops, bending — each causing different occlusions and path changes that static obstacles cannot replicate.

HA-R2R: Embedding Social Context into Route Descriptions

HA-R2R provides 16,844 curated navigation instructions generated with model assistance and human iterative revision. Instructions fuse human actions, group activities, and passing etiquette into routes. Average instruction length grows from 27 to 112 English words; vocabulary expands from 2,616 to 5,032.

Example: a living-room route may require avoiding a pacing phone caller, not disturbing a gathering group, and stopping at the door without blocking the entrance. Another route chains ironing and reading activities. Human descriptions serve as observable landmarks and give concrete behavioral boundaries to "pass" and "stop".

The test set contains 3,408 instructions from 18 held-out buildings, emphasizing multi-person routes. Holding out buildings tests generalization to new layouts rather than memorization of training rooms. Long instructions may present human cues at different stages, requiring models to retain execution progress and upcoming attention targets.

Navigation Interface and Baselines

The interface offers nearby human state queries, dynamic scene updates, and collision checking, enabling researchers to develop diverse strategies around the same scenes and isolate contributions of language understanding, visual perception, or action control.

Two reference agents are provided:

HA-VLN-VL : focuses on vision-language alignment.

HA-VLN-CMA : uses Cross-Modal Attention (CMA) to fuse vision, language, and action history, with training strategies improving adaptation to occlusions and execution drift.

Evaluation Results

On the continuous environment unseen validation split, BEVBert retrained on HA-VLN data achieves a success rate of 0.21 , while zero-shot transfer yields 0.15 . This gap shows the practical value of training with human activities and social instructions, yet highlights substantial room for improvement in dynamic navigation.

The paper reports navigation error, success rate, and two collision metrics side by side, emphasizing that proximity to the goal does not guarantee safer human interaction.

Real-Robot Validation

Methods are deployed on a Unitree Go2-EDU quadruped robot in office, living-room, corridor, and hall settings. The system demonstrates real-time human avoidance using the provided simulation interface and dynamic scene updates. Publicly released data, simulator, baselines, and evaluation interfaces allow future work to incrementally improve: understanding instructions with human activity, selecting appropriate routes in dynamic scenes, and minimizing human disturbance upon goal arrival.