How 30,000 Hours of Tactile Data Give Embodied AI a Human‑like Sense of Touch

The paper introduces a massive tactile‑visual dataset—NeoData—created by NeoteAI in collaboration with Fudan University to address the lack of a unified “dialect” for touch signals. The collection contains over 30,000 hours of interaction video, about 1.4 million operation clips (33 billion timesteps), 80 billion RGB frames and 100 billion tactile frames, captured with six robot platforms (Franka, Piper, UR5e, etc.) across 450 real‑world tasks performed by 90 operators. Five thousand hours of the data have already been open‑sourced.

To fuse heterogeneous touch signals, the authors propose the NeoForce unified representation model, which learns a transferable, temporally structured tactile embedding regardless of the underlying sensor hardware, effectively creating a “touch Mandarin” that enables cross‑device data integration.

Building on the dataset, the VTLA system augments visual‑language‑action (VLA) pipelines with tactile foresight. Instead of relying on delayed current touch feedback, VTLA predicts tactile evolution for the next 50 steps, encoding current touch into compact tokens and conditioning on visual context. In insertion tasks VTLA reaches 85 % success (vs. 60 % for vision‑only), and in key‑removal tasks 99 % success (vs. 35 %). VTLA also learns from failure data, using a progress‑assessment model and offline reinforcement learning to raise success rates on long‑horizon tasks such as towel folding (50 %→95 %), backpack packing (35 %→80 %), and box folding (20 %→75 %).

The TWAM framework extends this idea to world‑model construction, jointly predicting future video, tactile, and action sequences. TWAM employs a mixed‑expert architecture with three asynchronous expert networks (video, tactile, action); the video expert is warm‑started from a pretrained model, while tactile and action experts use narrower structures, resulting in 7.2 billion parameters—about half of a full‑width baseline. In simulation, TWAM achieves an average success rate of 84.5 % (baseline 36 %); on real robots it attains 46.3 % across eight tasks (LingBot‑VA baseline 21.9 %). Ablation studies confirm that removing either future tactile prediction or current tactile conditioning degrades performance sharply, underscoring tactile feedback’s indispensability in world models.

Collectively, the three technical reports argue that tactile modality exhibits a scaling potential comparable to vision: larger datasets and models yield systematic performance gains. The authors conclude that embodied AI is shifting from a purely visual paradigm to a “vision‑touch fusion” paradigm, where robots, like infants, actively explore through touch to validate hypotheses and plan actions, moving toward the long‑term goal of “touch‑aware” general intelligence.