Touch-R1: RL-Powered Tactile Reasoning Resolves Vision-Touch Conflicts (NeurIPS 2026)

Vision-Touch Conflict in Multimodal Models

Humans judge object physical properties by looking first, then touching to confirm or correct. Current multimodal large language models (MLLMs) heavily favor vision, which cannot perceive hardness, roughness, friction, deformation, or force, and fails on transparent, reflective, or occluded objects. Tests show that when vision and touch conflict, models like GPT-4o and Gemini-2.5-Pro trust visual appearance over tactile feel, leading to robot grasp failures, object damage, and safety risks.

Three Core Challenges for Tactile Reasoning

Scarcity of tactile data, especially tactile physical reasoning data.

Physical properties such as hardness and roughness are ordinal (degree-based) rather than categorical.

Understanding signals from different sensors while keeping judgments consistent across sensors.

TouchReason-1M: Large-Scale Cross-Sensor Tactile Reasoning Dataset

The team built TouchReason-1M , the first tactile data system for embodied models, fusing multi-sensor visuo-tactile data, ordinal attribute labels, and verifiable reasoning QA. It covers 1,000+ daily objects and 9 material categories , collected from four visuo-tactile sensors: DM-Tac X, GelSight Mini, Xense, and Tac3D . Unlike prior datasets, all objects were captured under strictly controlled standard motions and forces (press to 20N, then six exploration actions at standard force), enabling direct cross-object comparison. The dataset contains 14,700 valid tactile sequences and 132.3 million synchronized data pairs (raw tactile images, deformation, tangential change, depth change, and contact visual images), establishing a common physical reference for cross-sensor consistency learning.

Touch-R1 Model Architecture and Three-Stage Training

Touch-R1 adopts a three-stage training pipeline:

Future tactile frame prediction — learns dynamic contact laws.

Structured QA supervised fine-tuning — forms a "perception–comparison–conclusion" reasoning chain.

GRPO reinforcement learning tailored for touch — introduces:

Cross-sensor consistency reward to align physical attribute judgments across heterogeneous sensors.

Ordinal reward to capture degree relationships of hardness, roughness, etc.

Structured format reward and tactile input perturbation to force reasoning from tactile evidence, reducing shortcuts like guessing from vision alone.

TouchReason-Bench: Comprehensive Evaluation Benchmark

The benchmark comprises 4,800 QA pairs testing physical property identification, ordinal reasoning, comparative reasoning, cross-sensor consistency, and vision-touch conflict resolution, plus 200 unseen objects for generalization assessment.

Experimental Results

Touch-R1-7B achieves the best scores on all 9 core metrics , with an overall score of 60.1 .

Outperforms tactile specialist Octopi-13B by 18.4 points , GPT-4o by ~24.7 points , and previous strongest tactile model SToLa by 12.6 points .

Vision-touch conflict correction example : For an object that looks like hard plastic, Touch-R1 uses tactile deformation and force-depth curves to detect compliance, overturns the visual prior, and correctly classifies the material as rubber. This demonstrates the model's ability to decide which modality to trust when evidence conflicts.

Ablation studies confirm contributions from cold-start SFT, standard GRPO, ordinal reward, cross-sensor consistency constraint, and structured format control. Performance scales steadily with model size, indicating strong generalization potential.

Broader Impact

Touch-R1 shifts tactile perception from an auxiliary modality to a core modality for embodied foundation models. Its cross-sensor consistent physical reasoning enables transfer of tactile data and model capabilities across hardware platforms, providing foundational capabilities for real-world physical interaction tasks such as material identification, quality inspection, precision assembly, fragile object grasping, and deformable object manipulation.

References

Paper: https://arxiv.org/abs/2605.27154 (NeurIPS 2026). Project page: https://laiyingxin2.github.io/Touch-R1/. Code: https://github.com/laiyingxin2/Touch-R1. Model: https://huggingface.co/laiyingxin/Touch-R1. Dataset: https://huggingface.co/datasets/laiyingxin/TouchReason.