How TacForeSight Enables Robots to Anticipate Contact for Fine Manipulation

Background and Motivation

Robots can now see and feel, but reactive tactile feedback is insufficient for fine contact tasks such as wiping, card swiping, or insertion, where delays cause slipping or jamming. Predicting how contact will evolve enables proactive adjustment before errors occur.

Force‑Guided Tactile World Model (TacForceWM)

The core insight is that wrist force precedes fingertip tactile changes. TacForceWM encodes dual‑finger tactile fields into compact latent variables and uses high‑frequency wrist force/torque signals to forecast short‑term tactile evolution. The model operates in two coupled stages: (1) a force‑conditioned tactile world model predicts future tactile dynamics; (2) the predicted dynamics serve as a contact prior for a lightweight action‑generation policy.

This design avoids the computational cost of reconstructing high‑dimensional tactile images while preserving essential dynamic information, effectively learning "how current contact will evolve into future contact."

Predictive Tactile‑Conditioned Policy

After obtaining future tactile predictions, TacForeSight introduces a Predictive Tactile‑Conditioned Policy that treats the predicted tactile latent as a foresight prior. A cross‑attention mechanism explicitly models the relationship between current contact state and future trends, allowing the robot to consider both present and imminent contact during action generation. An adaptive gating mechanism dynamically balances visual and tactile contributions: tactile dominates during dense‑contact phases, while vision provides global context when the robot is away from contact.

Real‑World Experiments

Experiments were conducted on a platform equipped with a manipulator, gripper, camera, 6‑DoF force/torque sensor, and dual‑finger tactile sensor. Five representative contact‑intensive tasks were tested: vase wiping, card sliding, pipe insertion, bulb tightening, and flexible wire insertion, each under high‑disturbance conditions (height, angle, pose, lighting). TacForeSight achieved an average success rate of nearly 80% across the five standard tasks, outperforming pure‑vision models, simple vision‑touch‑force fusion, KineDex, FoAR, and RDP baselines. In dynamic‑disturbance scenarios, success rates reached 90% (height), 85% (angle), and 85% (pose), averaging 86.7%.

The system runs inference at 20 Hz, demonstrating that the model can be embedded in high‑frequency closed‑loop control and operate at speeds comparable to human manipulation.

Analysis of Learned Latent Variables

Visualization of the tactile latent space shows that, for bulb tightening and vase wiping, the predicted latent variables anticipate contact‑related changes about 200 ms earlier than the current tactile latent, confirming that the model captures temporal evolution rather than memorizing trajectories. t‑SNE clustering of unseen force‑touch interactions (pressing, twisting, sliding) reveals distinct, separable clusters, indicating the model’s ability to discriminate contact patterns and capture local deformation and force variations.

Implications

The work demonstrates that effective dexterous manipulation relies not on sensor quantity but on understanding the relational dynamics between modalities: wrist force provides a global leading signal, fingertip touch supplies fine‑grained feedback, and the world model bridges them into predictive contact dynamics. From the earlier OmniVTA framework to TacForeSight, the progression moves from "seeing" and "touching" to "foreseeing" the world, shifting from reactive feedback to proactive foresight and enabling real‑time, contact‑rich robot manipulation.