用视觉触觉联合模型生成真实感机器人操作轨迹，提升复杂抓取性能。
ViTacWorld: Scaling Visuo-Tactile World Models for Contact-Rich Robot Manipulation
- 融合真实与仿真数据构建视觉触觉世界模型，支持动作条件下的轨迹预测。
- 在接触密集任务中生成符合物理规律的模拟轨迹，提升策略性能23%以上。
- 适合需要触觉反馈的机器人抓取、装配等高精度操作场景研究者使用。
接触丰富的机器人操作需要感知摄像头无法捕捉的物理交互信号，触觉感知对实现稳健控制至关重要。然而，真实触觉交互数据收集成本高、依赖硬件且任务与场景多样性有限，导致视觉触觉学习难以规模化。本文提出ViTacWorld，一种面向接触丰富机器人操作的动作条件视觉触觉世界模型。该模型利用公开的真实触觉数据集和自建仿真环境，扩展视觉-触觉-动作数据，充分利用触觉信号直接反映物理接触的特性，其仿真到现实差距小于纯视觉观测。模型先在大规模真实与仿真轨迹上预训练，再通过真实策略回放微调以匹配下游操作行为。给定机器人动作，模型可预测时序对齐的视觉观测与触觉反馈，实现视觉触觉动作轨迹生成。据我们所知，这是首个使用世界模型进行机器人视觉触觉动作轨迹生成与策略评估的框架。其双重作用：一是合成轨迹用于增强下游触觉策略；二是通过预测受控动作序列下的视觉触觉结果，实现策略评估。在接触丰富操作任务上的实验表明，ViTacWorld能生成符合物理规律的轨迹，通过可扩展的数据增强提升策略性能，并支持动作条件下的策略评估。
原文摘要 · Abstract (English)
Contact-rich robot manipulation requires physical interaction cues that are often invisible to cameras, making tactile sensing essential for robust control. However, scaling visuo-tactile robot learning remains difficult because real tactile interaction data are expensive to collect, hardware-dependent, and limited in task and scene diversity. We present ViTacWorld, an action-conditioned visuo-tactile world model for scalable contact-rich robot manipulation. ViTacWorld leverages public real tactile datasets and a constructed simulation environment to scale visuo-tactile-action data, exploiting the fact that tactile signals are directly grounded in physical contact and can exhibit a smaller simulation-to-real gap than purely visual observations. The model is first pretrained with large-scale real and simulated visuo-tactile trajectories, and then finetuned with real-world policy rollouts to better match downstream manipulation behaviors. Given robot actions, ViTacWorld predicts temporally aligned visual observations and tactile feedback, enabling visuo-tactile-action rollout generation. To the best of our knowledge, ViTacWorld is the first framework that uses a world model for robot visuo-tactile-action trajectory generation and policy evaluation. It serves two roles: synthesizing rollouts to improve downstream tactile policies, and evaluating policies by predicting action-conditioned visuo-tactile outcomes under controlled action sequences. Experiments on contact-rich manipulation tasks show that ViTacWorld generates physically meaningful rollouts, improves policy performance through scalable data augmentation, and enables action-conditioned policy evaluation. Project page: https://vitacworld.github.io/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。