用视觉生成触觉信号,让机器人在无触觉传感器时也能精准抓握。
FELT: Generating Tactile Signals from Vision for Visuo-Tactile Manipulation

- 通过视觉编码器+轻量解码器,从图像直接生成手指触觉图。
- 在4个高接触任务中,生成触觉信号使成功率提升20%以上。
- 无需真实触觉传感器,适合缺乏触觉数据的机器人研究者。
触觉对操作至关重要,尤其在视觉受阻或模糊时。尽管视觉与触觉结合能提升操作性能,但训练鲁棒的跨模态策略需大量触觉数据,而触觉传感器脆弱、专用且难标准化,导致触觉数据远少于视觉数据。为此,我们提出特征提取的潜在触觉(FELT)框架,从RGB观测中合成单指压力触觉图像,减少对触觉设备数据采集的需求。FELT采用大型冻结视觉编码器和轻量查询解码器,在一次前向传播中预测触觉信号。为保留双指触觉传感器的物理拓扑结构,FELT通过独立分支解码左右传感器面板,捕捉擦拭、插入、手内旋转等交互中的非对称接触模式。推理时仅需RGB数据,可将现有纯视觉数据增强为触觉观测,既可用生成的触觉图像,也可用潜在触觉特征。四个高接触任务实验表明,生成触觉图像和潜在触觉特征均显著提升策略成功率,其中潜在特征在训练和部署阶段均无需真实触觉传感器。补充材料见:https://felt-tactile.github.io/。
原文摘要 · Abstract (English)
The sense of touch is central to manipulation, especially when vision is occluded or ambiguous. Although combining vision and touch improves manipulation, learning robust visuo-tactile policies requires substantial tactile data. Such data remains scarcer than visual data, because tactile sensors are fragile, specialized, and hard to standardize. To address this, we present Feature-Extracted Latent Tactile (FELT), a learning-based framework that synthesizes per-finger pressure tactile images from RGB observations, reducing the need for tactile-equipped data collection. FELT uses a large frozen visual encoder and a lightweight query decoder to predict tactile signals in a single feed-forward pass. To respect the physical topology of dual-finger tactile sensors, FELT decodes the left and right tactile sensor panels through separate branches, capturing the asymmetric contact patterns during interactions such as wiping, insertion, and in-hand rotation. At inference time, FELT only requires RGB data, allowing us to augment existing vision-only data with tactile observations, either as generated tactile images or as latent tactile features. Experiments on four contact-rich manipulation tasks demonstrate that both generated tactile images and latent tactile features improve policy success over vision-only baselines, with latent feature requiring no real tactile sensor during policy training or deployment. Supplementary material is available on our anonymous website: https://felt-tactile.github.io/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。