SimpleICL: Robots Learn New Tasks from One Demo Video Without Retraining

Traditional Vision-Language-Action (VLA) and World Action Model (WAM) policies rely on text instructions as task conditions, which cannot fully capture spatial relationships, interaction patterns, and execution details of manipulation tasks. Video demonstrations provide richer context, motivating the new paradigm of Robot In-context Learning (Robot ICL): a robot observes a human demonstration video to acquire task information and then generates actions conditioned on the current scene, achieving zero-shot generalization without per-task fine-tuning.

Core Problem: Prompt Ambiguity

A demonstration video simultaneously contains object semantics, action trajectories, manipulation affordances, and spatial relationships. Their importance varies per task, and the robot must know which information to attend to. SimpleICL formalizes this as Prompt Ambiguity and addresses it through three design steps.

Step 1: Define Learning Objectives

SimpleICL specifies that the robot must extract four core semantic categories from demonstrations: actions (what to do), objects (what to act on), task composition (sequential structure), and affordances (how to manipulate). Irrelevant factors — such as minor target-position changes, demonstrator-specific motion styles, lighting, background, and scene distractors — should not be mistaken for task semantics.

Step 2: Semantic Discriminative Data Construction

To teach the model to distinguish task-relevant from irrelevant cues, the authors design four discriminative data types:

Action discrimination : same scene and objects, different actions.

Object discrimination : target object position changed between demonstration and execution; similar distractor objects added to force tracking the target object rather than copying trajectories.

Composition discrimination : same action set arranged in different orders to teach temporal structure.

Affordance discrimination : same object manipulated in different ways (e.g., grasping a cup by body vs. handle) to learn fine-grained manipulation differences.

Additionally, irrelevant factors are augmented: different demonstrators, glove/skin colors, lighting, backgrounds, and camera viewpoints.

A low-cost grouped collection and cross-group pairing strategy is introduced. A "collection group" defines a set of executable tasks in one scene; after completion, object placements, lighting, or distractors are altered to create a new related scene. Human demonstrations from one group are paired with robot execution data from another group, preventing the model from associating demonstration-specific positions with execution targets. This yields ~3,000 valid human-robot pairs per day per platform.

Step 3: Simple Universal Demonstration Encoder

The demonstration video passes through a pretrained visual encoder to obtain frame-level features. A set of learnable query tokens uses cross-attention to extract task-relevant information from the entire video, compressing it into a fixed-length representation. This design supports arbitrary video lengths, integrates easily with existing policy backbones, and enables end-to-end training where the encoder optimizes jointly with the control objective.

Experiments

Simulation: 7 unseen tasks, SimpleICL achieves 66% average success rate vs. Fast-WAM 25%.

Real robot: 8 novel tasks under three difficulty settings (Easy, Medium, Hard). SimpleICL scores 81%, 70%, 68% respectively; FastWAM 42% and π0.5 31% on Hard.

Semantic discrimination analysis: Full data design yields 89% average intent-following rate across action, composition, and affordance discrimination. Composition is easiest, action discrimination moderate, affordance discrimination hardest and most dependent on targeted data.

Ablation on cross-group pairing: Under target position shift, success rate drops from 79% to 51% when cross-group pairing is removed, confirming its critical role in spatial generalization.

The real-robot data pipeline, training, and inference code are fully open-sourced, providing a reproducible foundation for further Robot ICL research.