arXiv:2607.23783cs.RO2026-07

首个大规模触觉驱动的机器人操作模型,能预测视觉、触觉与动作。

$N_0$-TWAM: Scaling Tactile-Native World-Action Model for Contact-Rich Manipulation

论文配图:$N_0$-TWAM: Scaling Tactile-Native World-Action Model for Contact-Rich Manipulation
图 1 · 摘自论文原文
  • 基于多模态数据预训练,融合六种机械臂与450个任务的触觉演示。
  • 在真实与仿真环境中均实现高精度触觉与动作预测,长时序任务表现优异。
  • 适合需要精细接触控制的机器人研究者,推动触觉智能发展。

我们提出 $N_0$-TWAM,一种面向高接触场景操作的大规模触觉原生世界-动作模型,可同时预测未来视觉与未来触觉。据我们所知,这是首个在大规模数据上训练的触觉世界-动作模型,在高接触任务中展现出强大能力。模型通过跨六种机器人形态、450项任务的丰富触觉演示进行视觉-触觉联合预训练。采用 NeoForce——一种统一的基于力的触觉表征——构建物理基础的接触信号,用于指导动作生成。为提升长时序与多阶段操作性能,引入触觉接触事件作为任务阶段标志,并在执行中逐步推进。为保证实时性,采用非对称混合变压器架构:全宽专家负责视频预测,轻量专家分别处理动作与触觉预测。在真实与模拟基准测试中,$N_0$-TWAM 在多种高接触任务中验证了其能力,表明数据规模扩展对精确触觉与动作预测的显著益处。综上,$N_0$-TWAM 使世界-动作模型具备预见视觉、触觉与动作的能力,为开放环境下的精细操作奠定了坚实基础。代码与模型权重将公开,以促进触觉增强型机器人操作的研究与发展。

原文摘要 · Abstract (English)

We present $N_0$-TWAM, a tactile-native world-action model for contact-rich manipulation that predicts both future vision and future contact. To our knowledge, it is the first tactile world-action model trained at large scale, and it shows strong capability on contact-rich tasks. We pre-train $N_0$-TWAM at large scale with visuo-tactile joint training over tactile-rich demonstrations spanning six embodiments and 450 tasks. We use NeoForce, a unified force-based tactile representation, to form a physically grounded contact signal that conditions action generation. To improve long-horizon and multi-stage manipulation, we introduce tactile contact events for task staging and advance through them during execution. For real-time efficiency, we adopt an asymmetric Mixture-of-Transformers architecture that pairs a full-width expert for video prediction with slim experts for downstream action and tactile prediction. Evaluations on both real and simulated benchmarks justify the capabilities of $N_0$-TWAM across a range of contact-rich tasks, and demonstrate the benefit of data scaling for precise tactile and action prediction. In summary, $N_0$-TWAM endows a world-action model with predictive capabilities to foresee vision, touch and action, building a solid foundation for fine-grained manipulation on open contact-rich tasks. The codebase and model checkpoints will be made publicly available to foster further research and development in tactile-enabled robotic manipulation.

触觉感知机器人操作世界模型多模态

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。