Predictive and Reactive Tactile Modeling: Making Robot Actions Truly Successful

Vision alone cannot confirm whether a robot has truly completed a manipulation; a robot may appear to press a button but fail to generate the expected effect because it lacks tactile confirmation of contact force.

TouchWorld addresses this gap by endowing tactile sensing with two complementary roles: a predictive component that forecasts the expected contact pattern before motion, and a reactive component that quickly corrects actions based on real‑time tactile feedback.

In the predictive stage, the model generates not only a future visual frame but also a tactile map that specifies which finger should apply pressure, where on the fingertip or palm the pressure should occur, and the approximate magnitude. This tactile map serves as a physical‑world target that the robot must achieve before the action is considered complete.

Because real‑world conditions introduce variations—objects may be offset, joints may drift from heat, or external disturbances may occur—re‑planning from scratch after every change would be too slow. TouchWorld therefore adds a reactive pathway that continuously reads the latest tactile signals and joint states at 30 Hz, applying fine‑grained adjustments such as slight lateral shifts, increased grip force, or wrist angle tweaks while the high‑level motion proceeds.

The system is organized into three hierarchical layers: a 1 Hz high‑level planner that decomposes tasks and predicts visual‑tactile goals; a 10 Hz mid‑level that generates the primary motion sequence; and a 30 Hz tactile‑feedback layer that performs rapid corrections.

TouchWorld builds on a data pipeline that starts with EgoTouch, a wearable capture system that synchronously records first‑person RGB video, dual‑hand 3D pose, and dense tactile glove data. EgoTouch covers 208 manipulation tasks, 1,891 interaction segments, over 20 hours of video, approximately 2.1 million frames, and more than 1,000 objects across diverse environments.

Using this dataset, the TouchAnything model learns a visual‑to‑tactile correspondence, enabling it to predict hand pressure distributions from ordinary videos. While it does not replace tactile hardware for fine manipulation, it provides a way to generate supervisory tactile signals for large‑scale video corpora.

TouchWorld is first pre‑trained on 20.2 hours of EgoTouch data and then fine‑tuned on 10 hours of robot demonstration data, aligning the learned tactile priors with the specific robot hand, sensor suite, and action space.

In six real‑robot tasks (watering, desk cleaning, plug insertion, cup insertion, pot wiping, tissue pulling), TouchWorld achieves an average success rate of 65.0% without disturbances and 57.2% with added target motion or grasp perturbations, surpassing the strongest baseline by 15.7 and 16.0 percentage points respectively. Ablation studies show that removing tactile input drops success to 43.3%/30.0%; removing the 30 Hz correction layer reduces it to 55.3%/40.3%; removing the tactile world model yields 60.2%/51.2%; and removing the sub‑task planner drops it to 55.5%/46.2%. A 4 B sub‑task planner with task‑stage supervision reaches 91% accuracy, outperforming a zero‑shot 32 B model at 84%.

These results indicate that simply scaling a single model is less effective than a hierarchical system that separates prediction and fast reaction, and that explicit tactile modeling substantially improves dexterous manipulation.

Beyond the model, the team emphasizes a full‑stack approach: from human data collection and tactile data cleaning, through visual‑tactile alignment and world‑model training, to remote‑operated dexterous hand control, sensor integration, and a unified evaluation framework. Future work includes open‑sourcing the EgoTouch, TouchAnything, and TouchWorld datasets, code, and models, and extending research toward whole‑body embodied manipulation.