让机器人预测触觉与视觉的交互未来,提升抓取规划成功率。
FeelWorld: Visuo-Tactile World Model for Hierarchical Contact Prediction and Planning

- 分层建模触觉状态,联合预测视觉与三类触觉信息。
- 80步滚动预测后,视觉误差降低61%,比纯视觉模型更稳定。
- 适合需要精准触觉反馈的机器人抓取与插入任务。
人类通过想象动作可能结果来规划物理交互,但现有视觉世界模型多关注外观变化,忽略影响接触密集型交互的触觉状态,可能导致看似合理却违背物理规律的预测。我们提出FeelWorld,一种分层的视觉-触觉世界模型,联合预测未来视觉隐变量和三种触觉状态:接触状态、3D触觉隐变量(编码力信息)以及滑动状态。这些状态由共享的隐动态模型联合预测,并获得显式监督。为避免自由空间运动中的无关触觉信号干扰视觉预测,引入接触门控的非对称注意力机制,使接触前保持纯视觉路径,接触时启用联合预测。模型还通过自回归滚动和上下文噪声注入训练以增强抗误差累积能力。预测出的接触与滑动状态支持接触感知的CEM规划。在芯片抓取、水果抓取和USB插入任务中,FeelWorld将10步LPIPS从0.084降至0.058,80步自回归滚动后LPIPS仍比视觉基线低61%。零样本规划成功率平均达81.7%,有效融合触觉感知于世界模型中。
原文摘要 · Abstract (English)
Humans plan physical interactions by imagining the possible outcomes of candidate actions. However, existing visual world models primarily capture appearance dynamics while overlooking the tactile states that govern contact-rich interactions, potentially producing imagined futures that appear visually plausible but violate physical dynamics. We introduce FeelWorld, a hierarchical visuo-tactile world model that jointly predicts future visual latents and three tactile states. FeelWorld organizes these states hierarchically as contact state, a 3D tactile latent that encodes force-related information, and slip state. These states are jointly predicted by a shared latent dynamics model with explicit supervision. To prevent irrelevant tactile signals during free-space motion from degrading visual prediction, we introduce a contact-gated asymmetric attention mechanism that maintains a visual-only prediction pathway before contact and enables joint visuo-tactile dynamics prediction during contact. The model is further trained with autoregressive rollouts and context noise injection to improve robustness to compounding errors. The predicted contact and slip states also support contact-aware CEM planning. Experiments on chip grasping, fruit grasping, and USB insertion show that FeelWorld reduces 10-step LPIPS from 0.084 to 0.058 and maintains an LPIPS that is 61% lower than that of the visual baseline after an 80-step autoregressive rollout. FeelWorld also achieves an average zero-shot planning success rate of 81.7%, providing an effective approach for incorporating tactile sensing into world models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。