用3D网格预测触觉，让机器人看一眼就能预判触摸感受。
FlowTouch: View-Invariant Visuo-Tactile Prediction
- 用物体局部3D网格编码触觉信息，摆脱视角依赖。
- 在仿真与真实场景间实现跨域泛化，支持新传感器部署。
- 生成的触觉图可用于判断抓取稳定性，适合机器人操控研究者。
触觉对接触密集型操作任务至关重要，能直接反馈物体几何、表面特性及作用力，提升感知并实现精细控制。触觉传感器的固有局限在于仅在接触时提供读数，无法用于任务规划和初始执行阶段。通过视觉预测触觉信息可弥补这一空白。传统方法是学习从相机图像到视觉触觉传感器输出的直接映射，但模型高度依赖特定设置，且受相机能否捕捉触碰区域的影响。本文提出FlowTouch，一种视图不变的视觉-触觉预测新模型。核心思想是利用物体的局部3D网格来编码丰富信息以预测触觉模式，同时抽象掉场景相关细节。FlowTouch融合场景重建与基于流匹配的图像生成模型。实验表明，FlowTouch能有效弥合仿真到真实世界的差距，并推广至新传感器实例。进一步验证了生成的触觉图像可用于下游抓取稳定性预测。代码、数据集和视频见https://flowtouch.github.io/
原文摘要 · Abstract (English)
Tactile sensation is essential for contact-rich manipulation tasks. It provides direct feedback on object geometry, surface properties, and interaction forces, enhancing perception and enabling fine-grained control. An inherent limitation of tactile sensors is that readings are available only when an object is touched. This precludes their use during planning and the initial execution phase of a task. Predicting tactile information from visual information can bridge this gap. A common approach is to learn a direct mapping from camera images to the output of vision-based tactile sensors. However, the resulting model will depend strongly on the specific setup and on how well the camera can capture the area where an object is touched. In this work, we introduce FlowTouch, a novel model for view-invariant visuo-tactile prediction. Our key idea is to use an object's local 3D mesh to encode rich information for predicting tactile patterns while abstracting away from scene-dependent details. FlowTouch integrates scene reconstruction and Flow Matching-based models for image generation. Our results show that FlowTouch is able to bridge the sim-to-real gap and generalize to new sensor instances. We further show that the resulting tactile images can be used for downstream grasp stability prediction. Our code, datasets and videos are available at https://flowtouch.github.io/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。