CUHK Survey: Harness Unifies Models, Data, Environment for Agentic Robots

Overview

The paper Towards Agentic Robots: Model, Data, Environment, and Harness (arXiv:2610.20442) from Hong Kong Chinese University MMLab and Fudan University presents a systematic survey of Agentic Robots —robots that use an agent to coordinate models, tools, and skills during task execution, adapt based on feedback, and accumulate reusable experience. The survey structures the field around four interconnected domains and 17 functional categories.

Core Motivation: Beyond Single Skills

Completing a task like “open drawer, take blue block, place on table” requires more than learning individual actions. The drawer may not open fully, the block may be occluded, or the grasp may fail. The robot must monitor progress, decide which capability to invoke next, and recover from errors. Agentic Robots address this by letting an agent orchestrate perception, planning, control, and skill execution in a closed loop.

Four Domains and 17 Functional Categories

1. Model (模型)

Covers how agents participate in training, perception, planning, robot control, and adaptation. Five sub‑categories:

Training – what to learn and how to update models.

Perception – understanding the current state (e.g., detecting drawer handle).

Planning – deciding high‑level steps (continue pulling vs. start grasping).

Robot Control – generating low‑level motor commands.

Adaptation – adjusting models based on experience (online or offline).

The survey distinguishes adjustments that serve only the current task (re‑observation, re‑planning) from those that affect future tasks (model retraining, skill library updates).

2. Data (数据)

Focuses on turning execution trajectories into useful training data through three stages:

Collection – deciding which tasks to attempt and what experience to gather.

Annotation – adding task descriptions, phase segmentation, and outcome labels.

Optimization – filtering effective segments, correcting noisy labels or supervision signals.

Agents can drive these decisions so that even failed episodes contribute valuable sub‑sequences (e.g., a successful drawer pull before a grasp failure).

3. Environment (环境)

Provides the physical or simulated settings for interaction and evaluation:

Environment Reconstruction – converting observed scenes into operable simulation assets (geometry, physics, articulation).

Task Generation – defining goals, constraints, and success conditions (which object, target pose, obstacles).

Evaluation Construction – specifying test scenarios, reset procedures, and metrics to measure generalization across object variations, layout changes, and disturbances.

Environment and data form a loop: environments enable interaction, interactions produce experience, data processing refines experience into training data, and independent evaluation validates capability gains.

4. Harness (Harness)

The software and mechanisms that coordinate models, tools, and robot interfaces during execution. Six functional groups:

Orchestration – scheduling component calls and handovers.

Memory – storing information for later decisions (e.g., past failures, object locations).

Monitoring & Recovery – checking task progress, detecting failures, triggering retries or fallbacks.

Skills – reusable operation primitives or callable programs (e.g., “open_drawer”).

Infrastructure – interfaces and runtime support between models and robot hardware.

Hierarchical Systems – coordinating high‑level reasoning, motion policies, and low‑level control.

Harness also preserves execution experience for reuse: a successful recovery can be distilled into a skill prompt or code patch without retraining the base model. The survey cautions that such updates must be validated separately for the current task and for future transfer.

Key Challenges for Real‑World Deployment

Fair Evaluation – success rates conflate policy strength, observation quality, and retry counts. Benchmarks must control these factors and report compute, interaction, and human‑assistance costs.

Real‑Robot Deployment – visual errors, reachability limits, and scene dynamics require agents to know skill preconditions, request additional observations, and verify execution state before proceeding.

Inference Efficiency – routine actions should run on lightweight skills with simple monitors; large models intervene only on anomalies. End‑to‑end latency (reasoning + communication + actuation + retries) must be measured.

Cross‑Embodiment Generalization – wheeled robots, arms, and drones have different sensors, action spaces, and motion constraints. Identifying which experience transfers and how to adapt it remains open.

Resources

Paper: https://arxiv.org/abs/2610.20442 Literature repository: https://github.com/chonghanqin/Awesome-Agentic-Robots The survey aims to clarify the agent’s role in robot systems—what decisions the agent makes, what the robot executes, and which experiences can be retained and reused—providing a unified framework for researchers and developers.