Can Tactile Sensing Complete Embodied AI's Perception of the Physical World?

Limitations of Vision‑Language‑Action (VLA) models for fine physical interaction

VLA models map visual and linguistic inputs directly to robot action sequences, enabling cross‑scene and cross‑object generalization through large‑scale vision‑language pre‑training [1-1]. However, they rely exclusively on non‑contact visual cues, which restricts them to coarse, low‑tolerance operations. Visual‑only world models can represent object geometry and appearance but cannot capture contact‑related mechanics such as material type, friction coefficient, or stiffness, preventing accurate manipulation in tasks that require precise force feedback [1-3].

Tactile sensing provides direct measurements of contact state, force feedback, and physical surface properties, complementing vision and enabling finer‑grained physical reasoning [1-3].

From hardware‑centric tactile research to algorithmic integration

Early tactile work focused on improving sensor precision and stability, but these hardware advances remained isolated from large‑model pipelines and did not influence task decision‑making or physical inference [1-3]. Recent research shifts toward algorithmic integration, aiming to standardize tactile representations, fuse them with multimodal models, and embed them within world‑model frameworks.

VTLA (Vision‑Tactile‑Language‑Action) adds a tactile branch to the VLA architecture. Visual, tactile, and language inputs are tokenized separately, then fed into a shared LLM Transformer. Multi‑head attention enables cross‑modal interaction, linking visual observations with tactile contact information to compensate for the visual‑only model’s lack of local touch perception [1-5].

N0‑VTLA / N0‑TWAM (NeoteAI + Fudan University) embed tactile temporal tokens at the tokenization stage, creating a multimodal paradigm where vision dominates and tactile data serves as a temporal supplement [1-4][1-5].

OmniVTLA (Shanghai Jiao‑Tong University + Paixun Intelligence) introduces a dual‑path tactile encoder and a semantic‑aligned tactile ViT (SA‑ViT) to achieve unified representation of heterogeneous visual and force‑based tactile signals.

Additional recent works— FTP‑1 , T‑Rex , TouchWorld , and Being‑H0.8 —explore multimodal joint reasoning, universal tactile representation construction, vision‑touch decoupled control, hierarchical tactile prediction, and implicit tactile pre‑training, respectively.

These approaches differ in modality‑fusion strategy, tactile‑representation paradigm, and dependence on specific hardware, reflecting a trade‑off between generality and sensor specificity.

Emerging direction

The field is moving from isolated tactile hardware toward comprehensive multimodal models that treat touch as an essential neural component, thereby bringing robots closer to reliable understanding and manipulation of the physical world.

Code example

1、近年来，视觉-语言-动作（VLA）模型成为具身智能的主流基础框架，依托大规模视觉-语言预训练，一定程度提升了机器人在非结构化环境中的任务适应性与长序指令执行能力。[1-1]
① 大规模视觉-语言预训练将高层语义理解与机器人动作指令对齐，实现了跨场景、跨物体的泛化操作能力。[1-1]
2、然而，面向精细物理交互场景，业界逐渐意识到 VLA 体系的局限性。研究者开始从多个方向拓展机器人的感知与控制能力，世界模型成为主流的优化方向，该模型弥补了传统 VLA 模型端到端映射、缺乏时序推演的缺陷。[1-2]
① 现有多数研究通过构建视觉世界模型，将其作为智能体的环境动力学模拟模块，依托视觉观测信息与候选动作序列，预测环境状态的时序演化过程，使机器人具备长时序规划能力。[1-2]
3、基于视觉的世界模型仅能表征物体空间几何结构与外观变化，无法捕捉接触力学相关信息，难以推断物体材质、摩擦系数、刚度等隐性物理属性，触觉模态的应用价值因而引起诸多关注。[1-3]
① 触觉可直接采集机器人与环境的接触状态、力学反馈及物体物理特征，能够补充视觉模态无法覆盖的物理交互信息，是完善机器人物理场景认知、支撑精细化交互任务的重要感知模态。[1-3]
4、相较于早期工作聚焦于触觉传感器的硬件研发与优化，近期工作着重于探索将触觉感知融入大模型与世界模型框架，实现触觉信息的标准化表征、跨模态融合与物理推理应用。[1-3]
① 早期触觉研究以硬件研发为主，重点提升触觉信号的采集精度与稳定性，但未接入大模型表征体系，也不参与智能体的任务决策与物理推理，仅作为独立的辅助传感模块使用，其交互信息的深层价值未得到挖掘。[1-3]