Agentic Robots: Models, Data, Environments, and Harness Unified

From Robot Policies to Agentic Robots

Traditional robot policies learn actions from demonstrations or reward signals. Vision-Language-Action (VLA) models integrate visual and linguistic knowledge into action generation. World models predict environment changes to support planning. World-action models further combine future prediction with action learning. However, possessing these capabilities is insufficient; the system must decide when to invoke a skill, how to combine skills, and how to adjust when execution fails. For example, in a task to open a drawer, grasp a blue block, and place it on a table, the drawer may open only halfway, the block may be occluded, or the grasp may fail. The Agent monitors the drawer state, judges whether grasping is possible, and decides whether to continue, re-observe, or switch strategies. The core of Agentic Robots is organizing and scheduling existing capabilities around a goal and dynamically adjusting execution strategies based on feedback. VLA, world models, and robot skills are all components of this system.

Four Domains: Model, Data, Environment, and Harness

The survey addresses both how robots complete immediate tasks and how the required capabilities are developed and iterated. Models support learning and decision-making, data provides usable experience, environments host practice and testing, and Harness coordinates the interplay among models, tools, and skills.

Model: Agent Participation in Learning, Decision-Making, and Control

The Model domain splits into five directions: training, perception, planning, robot control, and adaptation. Given the same pick-and-place failure, different root causes require Agent intervention at different stages. If the handle is not seen clearly, the Agent must acquire additional observations and understand spatial relations. If the drawer is not open when grasping starts, the Agent must adjust the task sequence. If the grasp pose is wrong, the Agent must correct the execution command. Improvements can also reach the learning layer: designing rewards, adjusting training regimens, or using experience to update model parameters and the information basis for decisions. Distinguishing these stages enables targeted failure analysis and selection of appropriate optimization methods.

Data: Making Every Attempt Leave Usable Experience

An incomplete task does not mean the entire data segment is worthless. The robot may have already opened the drawer or executed a successful recovery action after a grasp failure. Keeping only fully successful records wastes these valuable fragments. The Data domain covers collection, annotation, and optimization. The Agent can arrange attempts across different tasks and initial states, supplement task descriptions, action phases, and outcome labels, then filter effective segments and correct erroneous annotations. For instance, separately labeling "open drawer", "grasp failure", and "retry after adjustment" makes it easier to identify experience usable for learning.

Environment: Where to Practice, What to Practice, How to Test

Robots need environments for practice and capability verification. A simulated drawer that only looks realistic is insufficient; handle position, joint motion, and object placement all affect manipulation outcomes. The survey explores three directions: environment reconstruction, task generation, and evaluation construction. The Agent can help build interactive scenes, set goals, constraints, and success conditions, and design test scenarios by moving objects or adding obstacles to verify whether the robot adapts to new situations.

Harness: Coordinating Capabilities to Complete Tasks

Individual components may work in isolation, but integration points remain error-prone. A program returning "execution completed" does not guarantee the drawer actually opened; an object may have just moved while the decision system still relies on stale information. Harness manages observation information, system state, and task progress, coordinates models, tools, and skills, and validates execution results. In the blue-block task, Harness first confirms drawer state and grasp preconditions, then the Agent decides whether to continue, acquire more observations, or switch skills. Planning, execution, and low-level control must collaborate at their respective operating rhythms. Experience can be stored in a memory bank, validated operations can be distilled into reusable skills, and execution code can be optimized based on feedback without retraining the model. However, whether these changes truly improve capability must be verified on new tasks while ensuring existing capabilities do not regress.

Conclusion

From deciding the next action to preparing data, building environments, and orchestrating skill coordination, Agents are permeating every stage of robot capability development and task execution. The full survey and accompanying literature repository are available for further study, and the authors welcome discussion on how robots can complete tasks more reliably and iterate capabilities through experience.

https://arxiv.org/abs/2610.20442
https://github.com/chonghanqin/Awesome-Agentic-Robots