A Comprehensive Survey of Tactile‑Based Multimodal Fusion in Embodied Intelligence

Why tactile is essential for embodied intelligence

Touch provides direct, near‑field feedback about surface geometry, material properties, and contact dynamics that remote sensors such as vision cannot capture. This feedback closes the perception‑action loop, enabling precise manipulation and stable grasping in physically interactive environments.

Four‑stage tactile fusion pipeline

Physical transduction & spatiotemporal observation – sensors convert deformation, force, or vibration into digital signals (e.g., high‑dimensional tensors or image streams).

Modality‑specific representation learning – dedicated encoders (e.g., ResNet or ViT for vision/tactile, OpenCLIP for language) map each modality to a unified latent vector.

Cross‑modal fusion – feature concatenation, cross‑attention, or contrastive alignment produce a shared joint representation.

Embodied decoding & task execution – the fused representation is decoded into downstream outputs such as object class, textual description, or robot control actions.

Multimodal tactile dataset taxonomy

Datasets are grouped by modality composition:

T‑V (tactile‑vision) : early VT Dataset (controlled grasping) → Touch in the Wild (outdoor scenes) → TouchClothing (deformable objects).

T‑L (tactile‑language) : PhysiCLEAR (hardness, roughness) → STOLA (open‑ended tactile commonsense reasoning).

T‑V‑L (tactile‑vision‑language) : Touch100k (>100 k aligned tri‑modal samples with short tags and long natural‑language descriptions).

T‑V‑O (tactile‑vision‑other) : ObjectFolder series (adds impact audio) → OmniViTac (adds action sequences for end‑to‑end manipulation learning).

Method paradigms

Multimodal perception & recognition

Multimodal object recognition – early feature concatenation and recent Transformer‑based joint queries (e.g., VHTformer) enable recognition under visual ambiguity such as transparent objects.

Attribute & material identification – progression from supervised classifiers to zero‑shot CLIP‑style models (e.g., UniTouch) that infer material from language prompts.

Grasp success/failure prediction – uses post‑contact tactile cues (slippage, force distribution) for closed‑loop stability assessment.

Cross‑modal retrieval & matching – evaluates representation alignment by retrieving visual images or textual descriptions from tactile inputs.

Cross‑modal generation & translation

Vision‑to‑tactile and tactile‑to‑vision synthesis – generate tactile deformation maps from rock photos or reconstruct visual textures from touch data.

Language‑to‑tactile translation – tactile caption generation (e.g., VTV‑LLM) and text‑to‑tactile synthesis that creates touch data from textual descriptions.

Multimodal interaction & manipulation

Perception‑driven robot manipulation – tactile feedback guides fine assembly (e.g., insertion tasks) and stable grasping; DexTac demonstrates high‑precision syringe insertion using contact‑region cues.

Language‑conditioned multimodal operation – large‑language‑model‑augmented VLA systems interpret abstract commands (e.g., “gently grasp the soft object”) by jointly reasoning over language, vision, and tactile streams to generate continuous actions.

Hardware landscape

Wearable tactile systems – capture human interaction priors for data collection and skill transfer.

Hand‑held and fingertip sensors – provide high‑resolution local contact perception, suitable for direct integration on robot end‑effectors.

Robotic skin & multimodal patches – large‑area, compliant, distributed sensing for whole‑body perception.

Gripper‑mounted integrated sensors – embed perception at the manipulation interface for compact, co‑located feedback.

Evaluation gaps and benchmark challenges

Data fragmentation & scalability bottlenecks – task‑specific, sensor‑dependent datasets are far smaller than vision‑language resources, limiting zero‑shot transfer.

Modal misalignment & noise – sparse tactile streams misalign with dense visual/language inputs; sensor drift and visual occlusion further degrade alignment.

Hardware‑software integration barriers – heterogeneous sensor form‑factors lack standard interfaces; power and durability constraints hinder real‑time fusion with large models.

Inconsistent evaluation metrics – benchmarks are task‑centric, lacking end‑to‑end embodied metrics for safety, robustness, and control effectiveness.

Challenges and future directions

Key research directions include:

Building extensible, large‑scale multimodal datasets to match the data demands of large language models.

Evolving hierarchical fusion architectures that treat tactile as a foundational reasoning modality.

Developing durable, edge‑processed tactile skins to expand perception boundaries.

Embedding tactile feedback as continuous supervision in decision loops to enable transition from controlled labs to complex real‑world environments.

GitHub repository: https://github.com/Wayne-coding/Multimodal-Tactile-Sensing-and-Fusion

Code example

触觉 - 视觉 (T-V) 数据集： 早期（如 VT Dataset）主要关注受控环境下的机器人抓取；近期则向无约束的野外环境（如 Touch in the Wild）和复杂形变物体（如 TouchClothing）发展。
触觉 - 语言 (T-L) 数据集： 旨在建立触觉与人类认知的桥梁。例如 PhysiCLEAR 记录了物体的软硬、粗糙度，而最新的 STOLA 则支持开放式的触觉常识推理，打破了过去只能依赖视觉进行语义接地的局限。
触觉 - 视觉 - 语言 (T-V-L) 数据集： 迎合当前大模型趋势的终极形态。比如 Touch100k 包含了超 10 万个三模态对齐样本，不仅有短语标签，还有长文本自然语言描述，完美支持跨模态对齐。
触觉 - 视觉 - 其他 (T-V-O) 数据集： 引入了动作、音频或本体感觉。著名的 ObjectFolder 系列结合了撞击音频；而 OmniViTac 等数据集则加入了动作序列，支持端到端的接触丰富型操作策略学习。