How Robots Turn World Representations into Action: Jiajun Wu's ECCV 2026 Insights

Introduction

At ECCV 2026 in Malmö, Stanford assistant professor Jiajun Wu delivered a talk titled Physical AI: Understanding and Building the Physical World . The central question: given a structured representation of the world, how can a robot use it to act reliably in open-ended, unpredictable environments? Wu's presentation traced a line from his IJCAI-ECAI 2026 award lecture on Building Visual and Physical Intelligence Through Code — which introduced the notion of physical code (geometry, material, affordances, physics) — to the next step: turning that understanding into action.

The Robot's Challenge: Beyond Pixel Recognition

Wu illustrated the gap with a kitchen scenario: a robot must wash a dish that has detergent on it, but the human leaves, returns, puts the clean dish back, or the dish has trash underneath. The robot must not only execute a fixed sequence but continuously replan: detect state changes (clean vs. dirty), infer preconditions (hold before wash, faucet on), and compose atomic skills (pick, wash, place) into longer horizons. This requires compositional abstractions that generalize across objects, environments, and human behaviors.

Compositional Abstractions for Long-Horizon Tasks

The proposed paradigm learns compositional abstractions for inference end-to-end from raw supervision (natural language, images, video). From a teleoperated demonstration video, a caption model generates subtitles. The learning algorithm discovers:

States — binary predicates verified by neural networks (e.g., is_dirty(obj), on(objA, objB)).

Actions (atomic skills) — e.g., wash requires hold and faucet_on; postcondition: object becomes clean if detergent present.

Relations — symbolic types defining compositionality; neural networks handle perception and control.

This structure enables forward reasoning: if preconditions hold, what are the effects? What actions become permissible? Crucially, the predicates and skills are discovered by foundation models, not hand-coded. A verifier network checks consistency with the demonstration. Wu likened this to Claude Code in the digital realm: decomposing a complex task into verified steps with pre/post conditions.

The system can be assembled from off-the-shelf models (VLMs for abstraction proposal, perception models for state verification) and supports fine-tuning for novel objects (e.g., a never-before-seen factory part). A key limitation of the first version: all verification occurred in 2D, insufficient for spatial reasoning like on(objA, objB). The extension ingests 3D point-cloud trajectories over time, enabling detection, state verification, and relation checking in 3D, and demonstrating generalization to longer horizons (e.g., placing two books after training on one).

Neural Kinematics: Predicting Interactions Without Demonstrations

When no human demonstration exists, how can a robot infer how to interact with a novel object? Wu's Neural Kinematics (ECCV 2026) captures an object's intrinsic kinematic space — the manifold of its possible deformations — from a static mesh. The framework uses an encoder-decoder with a conditional VAE :

Encoder 1 (conditional): mesh + deformation field → latent distribution.

Encoder 2 (mesh-only): mesh → latent distribution.

Losses: reconstruction loss (align with simulated demonstrations) + KL divergence between the two latent distributions.

At test time, only the mesh-only encoder is used; sampling yields plausible deformations. The method achieves zero-shot generalization to unseen objects (faucet, Newton's cradle, lamp, laptop, chair, cabinet) and even to noisy iPhone 3D scans with heavy artifacts, still predicting plausible interaction modes. These predicted interaction sequences serve as free demonstrations for downstream skill learning.

Distilling Video Diffusion Models for Free Demonstrations

A third route leverages video diffusion models as priors. Starting from a static 3D scene (represented as 3D Gaussian Splatting with an added time dimension → 4D representation), the system renders multi-view videos, computes a score from a video diffusion model conditioned on a language prompt (e.g., a robot arm picks up a brick), and backpropagates gradients to optimize the 4D representation — a generalized Score Distillation Sampling (SDS) in 4D. This yields dense point trajectories and realistic interaction sequences for objects like a basketball hitting a rim, a cat jumping on a cushion, or a cloth being manipulated. A real robot then learns the task from these generated demonstrations without any human teleoperation or video collection. Wu noted the approach has not yet been tested on truly novel object categories beyond the training distribution of the video model.

Benchmarks for Embodied Reasoning

Wu outlined three benchmarks to evaluate structured representations:

Inact (CoRL 2026): Factorizes reasoning in a POMDP-inspired setup. Given initial state, state evolution, and a set of candidate actions, the model must reorder actions to explain the observed state changes — avoiding the difficulty of direct action prediction in huge action spaces.

Embodied Spatial Intelligence Benchmark : An active agent that can move (e.g., move_forward) to answer questions (3D geometry, counting, layout, goal-directed actions) across 10 categories and ~29 subcategories. The agent must sometimes navigate to observe occluded objects.

Household Activities Benchmark : A six-year effort simulating 1,000 daily activities derived from U.S. Bureau of Labor Statistics surveys and user benefit ratings (top: clean room after party , clean shower ). First challenge (2025) used 50 tasks, 3,000 hours of human demonstrations, 20+ teams. Second challenge adds more tasks and demonstrations.

Closing Remarks and Q&A

Wu emphasized that the structure of representations (geometry, task dependencies) is largely unified; the open questions are (1) what exactly that unified representation is, (2) how to represent it, and (3) how to acquire it. In Q&A, he clarified that their video diffusion distillation does not generate videos but uses the model as a score-based prior (like SDS). On implicit vs. explicit representations (e.g., Dreamer), he acknowledged latent-space planning is promising but current world action models are more impressive. On reward model staleness, he suggested large foundation models as rewards are less prone to obsolescence than small metrics like FID, though human-in-the-loop data collection remains a bottleneck.